I recently deployed Burp Enterprise using Portswigger's Kubernetes Helm Chart, and I encountered an error when launching the "Pre-Scan Check". This feature uses a Chrome browser to verify network connectivity to the targeted site, and it verifies the recorded login sequence can log in successfully in preparation for running a scan. When I used this feature, the "Pre-Scan Check" would partially complete and then just stop working. I spent the next couple hours digging into how it worked and trying to figure out a solution.
First, let's start by talking about the installation. When you install the Burp Enterprise Kubernetes Helm Chart, it creates (among other things) a deployment for the bsee-web-server, bsee-enterprise-server, and matching services. You can customize the installation of these items by modifying the values.yaml file. I installed this chart initially with very few changes to that file.
Next, I logged into the Burp Enterprise web interface and added a site, recorded a login sequence, and configured a scan. Before launching the scan, I hit the "Pre-Scan Check" button and it mysteriously quit working part way through. To troubleshoot, I ran the "Pre-Scan Check" again and ran k -n burp get all and identified that a job named bsee-connection-check-2 was created to complete the "Pre-Scan Check" tasks. After watching that job with k -n burp get jobs -o wide -w, I noticed that after a few minutes, the job would terminate with a status of OOMKilled. This error indicates that the job exceeded the memory limit for the pod which was initially set as 1Gi.
This setting can be modified in the values.yaml file, so I increased it by modifying connectionCheckContainerMemory: 1Gi to connectionCheckContainerMemory: 2Gi and ran helm upgrade to persist that change. After running the "Pre-Scan Check" again, the job still died with an OOMKilled status. When I ran k -n burp describe job bsee-connection-check-3, it stated that the memory limit was still 1Gi. I was quite confused by this.
I spent some time hunting for the bsee-connection-check job template and had a hard time finding it. Eventually, i figured out where it was by running a helm -n burp upgrade bsee ./burphelm -f values.yaml --dry-run. This command prints out the yaml files that will be applied as part of the upgrade. Within those yaml files I found the BSEE_CONNECTION_CHECK_JOB_TEMPLATE in the following ConfigMap:
# Source: burp-suite-enterprise-edition/templates/enterprise-server/config-map.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: bsee-enterprise-server-config
namespace: burp
labels:
app.kubernetes.io/name: enterprise-server
app.kubernetes.io/component: server
app.kubernetes.io/part-of: bsee
app.kubernetes.io/instance: bsee
app.kubernetes.io/version: 2023.2-12088
app.kubernetes.io/managed-by: Helm
helm.sh/chart: burp-suite-enterprise-edition-2023.2.0
data:
BSEE_HTTPS_KEYSTORE_LOCATION: /home/burpsuite/keystores/es.keystore
BSEE_HTTPS_PORT: "8072"
BSEE_LINUX_PREFS_STORE_PATH: /home/burpsuite/prefs
BSEE_DATA_PATH: /home/burpsuite/data
BSEE_BURP_DOWNLOAD_PATH: /home/burpsuite/burp
BSEE_LOGS_DIRECTORY: /home/burpsuite/logs
BSEE_SCAN_JOB_TEMPLATE: |-
YXBpVmVyc2lvbjogYmF0Y2gvdjEKa2luZDogSm9iCm1ldGFkYXRhOgogIGxhYmVsczoKICAgIGFwcC5rdWJlcm5ldGVzLmlvL2NvbXBvbmVudDogc2Nhbm5lcgogICAgYXBwLmt1YmVybmV0ZXMuaW8vcGFydC1vZjogYnNlZQpzcGVjOgogIGJhY2tvZmZMaW1pdDogMAogIHRlbXBsYXRlOgogICAgbWV0YWRhdGE6CiAgICAgIGxhYmVsczoKICAgICAgICBhcHAua3ViZXJuZXRlcy5pby9jb21wb25lbnQ6IHNjYW5uZXIKICAgICAgICBhcHAua3ViZXJuZXRlcy5pby9wYXJ0LW9mOiBic2VlCiAgICAgIGFubm90YXRpb25zOgogICAgICAgIGNsdXN0ZXItYXV0b3NjYWxlci5rdWJlcm5ldGVzLmlvL3NhZmUtdG8tZXZpY3Q6ICJmYWxzZSIKICAgIHNwZWM6CiAgICAgIGNvbnRhaW5lcnM6CiAgICAgICAgLSBlbnY6CiAgICAgICAgICAgIC0gbmFtZTogQlNFRV9FUEhFTUVSQUxfQUdFTlRfTElDRU5TSU5HX01BWF9SRVRSSUVTCiAgICAgICAgICAgICAgdmFsdWU6ICI1IgogICAgICAgICAgICAtIG5hbWU6IEJTRUVfRVBIRU1FUkFMX0FHRU5UX0xJQ0VOU0lOR19SRVRSWV9ERUxBWQogICAgICAgICAgICAgIHZhbHVlOiAiMTIwIgogICAgICAgICAgICAtIG5hbWU6IEJTRUVfTE9HU19ESVJFQ1RPUlkKICAgICAgICAgICAgICB2YWx1ZTogL2hvbWUvYnVycHN1aXRlL2xvZ3MKICAgICAgICAgICAgLSBuYW1lOiBCU0VFX0JVUlBfRE9XTkxPQURfUEFUSAogICAgICAgICAgICAgIHZhbHVlOiAvaG9tZS9idXJwc3VpdGUvYnVycAogICAgICAgICAgICAtIG5hbWU6IEVQSEVNRVJBTF9BR0VOVF9PUFRTCiAgICAgICAgICAgICAgdmFsdWU6IC1YbXMxMjhtIC1YbXg1MTJtCiAgICAgICAgICAgIC0gbmFtZTogQlNFRV9CVVJQX0pBVkFfT1BUUwogICAgICAgICAgICAgIHZhbHVlOiAtWG1zMWcgLVhYOk1heFJBTVBlcmNlbnRhZ2U9NzUKICAgICAgICAgICAgLSBuYW1lOiBCU0VFX0VQSEVNRVJBTF9TRVRUSU5HU19QQVRICiAgICAgICAgICAgICAgdmFsdWU6IC9ob21lL2J1cnBzdWl0ZS9zZWNyZXRzL3NjYW5Jbml0aWF0aW9uUmVxdWVzdC5qc29uCiAgICAgICAgICBpbWFnZTogcHVibGljLmVjci5hd3MvcG9ydHN3aWdnZXIvZXBoZW1lcmFsLWFnZW50OjIwMjMuMi0xMjA4OAogICAgICAgICAgaW1hZ2VQdWxsUG9saWN5OiBJZk5vdFByZXNlbnQKICAgICAgICAgIHJlc291cmNlczoKICAgICAgICAgICAgbGltaXRzOgogICAgICAgICAgICAgIGNwdTogMTQwMG0KICAgICAgICAgICAgICBtZW1vcnk6IDhHaQogICAgICAgICAgICByZXF1ZXN0czoKICAgICAgICAgICAgICBjcHU6IDE0MDBtCiAgICAgICAgICAgICAgbWVtb3J5OiA4R2kKICAgICAgICAgIHZvbHVtZU1vdW50czoKICAgICAgICAgICAgLSBtb3VudFBhdGg6IC9ob21lL2J1cnBzdWl0ZQogICAgICAgICAgICAgIG5hbWU6IGJzZWUtdm9sdW1lCiAgICAgIHJlc3RhcnRQb2xpY3k6IE5ldmVyCiAgICAgIHRlcm1pbmF0aW9uR3JhY2VQZXJpb2RTZWNvbmRzOiAzMAogICAgICB2b2x1bWVzOgogICAgICAgIC0gbmFtZTogYnNlZS12b2x1bWUKICAgICAgICAgIHBlcnNpc3RlbnRWb2x1bWVDbGFpbToKICAgICAgICAgICAgY2xhaW1OYW1lOiBidXJwLXB2Yw==
BSEE_CONNECTION_CHECK_JOB_TEMPLATE: |-
YXBpVmVyc2lvbjogYmF0Y2gvdjEKa2luZDogSm9iCm1ldGFkYXRhOgogIGxhYmVsczoKICAgIGFwcC5rdWJlcm5ldGVzLmlvL2NvbXBvbmVudDogc2Nhbm5lcgogICAgYXBwLmt1YmVybmV0ZXMuaW8vcGFydC1vZjogYnNlZQoKc3BlYzoKICBiYWNrb2ZmTGltaXQ6IDAKICB0ZW1wbGF0ZToKICAgIG1ldGFkYXRhOgogICAgICBsYWJlbHM6CiAgICAgICAgYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50OiBzY2FubmVyCiAgICAgICAgYXBwLmt1YmVybmV0ZXMuaW8vcGFydC1vZjogYnNlZQogICAgICBhbm5vdGF0aW9uczoKICAgICAgICBjbHVzdGVyLWF1dG9zY2FsZXIua3ViZXJuZXRlcy5pby9zYWZlLXRvLWV2aWN0OiAiZmFsc2UiCiAgICBzcGVjOgogICAgICBjb250YWluZXJzOgogICAgICAgIC0gZW52OgogICAgICAgICAgICAtIG5hbWU6IEJTRUVfRVBIRU1FUkFMX0FHRU5UX0xJQ0VOU0lOR19NQVhfUkVUUklFUwogICAgICAgICAgICAgIHZhbHVlOiAiNSIKICAgICAgICAgICAgLSBuYW1lOiBCU0VFX0VQSEVNRVJBTF9BR0VOVF9MSUNFTlNJTkdfUkVUUllfREVMQVkKICAgICAgICAgICAgICB2YWx1ZTogIjEyMCIKICAgICAgICAgICAgLSBuYW1lOiBCU0VFX0xPR1NfRElSRUNUT1JZCiAgICAgICAgICAgICAgdmFsdWU6IC9ob21lL2J1cnBzdWl0ZS9sb2dzCiAgICAgICAgICAgIC0gbmFtZTogQlNFRV9CVVJQX0RPV05MT0FEX1BBVEgKICAgICAgICAgICAgICB2YWx1ZTogL2hvbWUvYnVycHN1aXRlL2J1cnAKICAgICAgICAgICAgLSBuYW1lOiBFUEhFTUVSQUxfQUdFTlRfT1BUUwogICAgICAgICAgICAgIHZhbHVlOiAtWG1zMTI4bSAtWG14NTEybQogICAgICAgICAgICAtIG5hbWU6IEJTRUVfQlVSUF9KQVZBX09QVFMKICAgICAgICAgICAgICB2YWx1ZTogLVhtczFnIC1YWDpNYXhSQU1QZXJjZW50YWdlPTc1CiAgICAgICAgICAgIC0gbmFtZTogQlNFRV9FUEhFTUVSQUxfU0VUVElOR1NfUEFUSAogICAgICAgICAgICAgIHZhbHVlOiAvaG9tZS9idXJwc3VpdGUvc2VjcmV0cy9jb25uZWN0aW9uQ2hlY2tJbml0aWF0aW9uUmVxdWVzdC5qc29uCiAgICAgICAgICBpbWFnZTogcHVibGljLmVjci5hd3MvcG9ydHN3aWdnZXIvZXBoZW1lcmFsLWFnZW50OjIwMjMuMi0xMjA4OAogICAgICAgICAgaW1hZ2VQdWxsUG9saWN5OiBJZk5vdFByZXNlbnQKICAgICAgICAgIHJlc291cmNlczoKICAgICAgICAgICAgbGltaXRzOgogICAgICAgICAgICAgIGNwdTogMTQwMG0KICAgICAgICAgICAgICBtZW1vcnk6IDJHaQogICAgICAgICAgICByZXF1ZXN0czoKICAgICAgICAgICAgICBjcHU6IDE0MDBtCiAgICAgICAgICAgICAgbWVtb3J5OiAyR2kKICAgICAgICAgIHZvbHVtZU1vdW50czoKICAgICAgICAgICAgLSBtb3VudFBhdGg6IC9ob21lL2J1cnBzdWl0ZQogICAgICAgICAgICAgIG5hbWU6IGJzZWUtdm9sdW1lCiAgICAgIHJlc3RhcnRQb2xpY3k6IE5ldmVyCiAgICAgIHRlcm1pbmF0aW9uR3JhY2VQZXJpb2RTZWNvbmRzOiAzMAogICAgICB2b2x1bWVzOgogICAgICAgIC0gbmFtZTogYnNlZS12b2x1bWUKICAgICAgICAgIHBlcnNpc3RlbnRWb2x1bWVDbGFpbToKICAgICAgICAgICAgY2xhaW1OYW1lOiBidXJwLXB2Yw==
When that value is base64 decoded, it yields:
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: scanner
app.kubernetes.io/part-of: bsee
spec:
backoffLimit: 0
template:
metadata:
labels:
app.kubernetes.io/component: scanner
app.kubernetes.io/part-of: bsee
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
containers:
- env:
- name: BSEE_EPHEMERAL_AGENT_LICENSING_MAX_RETRIES
value: "5"
- name: BSEE_EPHEMERAL_AGENT_LICENSING_RETRY_DELAY
value: "120"
- name: BSEE_LOGS_DIRECTORY
value: /home/burpsuite/logs
- name: BSEE_BURP_DOWNLOAD_PATH
value: /home/burpsuite/burp
- name: EPHEMERAL_AGENT_OPTS
value: -Xms128m -Xmx512m
- name: BSEE_BURP_JAVA_OPTS
value: -Xms1g -XX:MaxRAMPercentage=75
- name: BSEE_EPHEMERAL_SETTINGS_PATH
value: /home/burpsuite/secrets/connectionCheckInitiationRequest.json
image: public.ecr.aws/portswigger/ephemeral-agent:2023.2-12088
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1400m
memory: 2Gi
requests:
cpu: 1400m
memory: 2Gi
volumeMounts:
- mountPath: /home/burpsuite
name: bsee-volume
restartPolicy: Never
terminationGracePeriodSeconds: 30
volumes:
- name: bsee-volume
persistentVolumeClaim:
claimName: burp-pvc
Jackpot! The job template is passed in a ConfigMap as an environment variable to the bsee-enterprise-server as shown in the snippet below:
containers:
- name: enterprise-server
image: public.ecr.aws/portswigger/enterprise-server:2023.2-12088
resources:
requests:
cpu: 1400m
memory: 4Gi
limits:
cpu: 4000m
memory: 8Gi
env:
- name: BSEE_INSTALLATION_ENVIRONMENT
value: KUBERNETES
- name: ENTERPRISE_SERVER_OPTS
value: -Xms1g -Xmx5g
- name: BSEE_BURP_JAVA_OPTS
value: -Xms128m -Xmx512m
- name: BSEE_ENTERPRISE_SERVER_SERVICE_HOST
value: $(BSEE_ENTERPRISE_SERVER_SERVICE_HOST)
- name: BSEE_ENTERPRISE_SERVER_SERVICE_PORT_EPHEMERAL_AGENT
value: $(BSEE_ENTERPRISE_SERVER_SERVICE_PORT_EPHEMERAL_AGENT)
- name: BSEE_MUTABLE_CONFIG_REF
value: bsee-enterprise-server-config
- name: BSEE_MUTABLE_SECRET_REF
value: bsee-enterprise-server-secret
- name: BSEE_CONTAINER_REGISTRY
value: public.ecr.aws
envFrom:
- configMapRef:
name: bsee-enterprise-server-config
Since its added to the container through an environment variable instead of a volume mount, the enterprise server has to be restarted for it to pick up the new environment variable and job template. In addition, a helm upgrade will not trigger a restart of the server, because I didn't change the bsee-enterprise-server deployment file itself. The final solution was to run k -n burp rollout restart deployment bsee-enterprise-server. Afterward, the bsee-connection-check jobs all picked up the 2Gi of memory and ran to completion.
Posts
No comments:
Post a Comment