I recently figured out a lingering issue with my Gitlab setup that has been an annoyance for a few weeks now. A few weeks ago, my Gitlab stopped loading content when I would open a project. The message said something along the lines of “Error fetching folder content.” For a while, I was too busy with everything I’ve got going on to sit down and finally figure it out. A couple days ago I finally had a while blocked out to sit down and find the issue, and I finally fixed it. I’ve decided to document my hunt to serve as a reminder for myself and a learning experience for others.
My Gitlab server isn’t on the same host as my web server, so I have my web server configured to connect requests to Gitlab via reverse proxy. When I would browse directly to Gitlab while on the local network, I wouldn’t get the error I described. I figured it had to be something wrong with my Apache reverse proxy configuration or Gitlab’s backend configuration for using a reverse proxy. I scoured documentation for a couple of hours to make sure I fully understood each part of my configuration files. When I was sure that neither end’s config needed to be changed, I decided to dive into networking artifacts from the requests.
I opened up developer tools on Chrome and noticed that when I was trying to open a Gitlab project externally, I was getting a 403 forbidden error on a POST request to /api/graphql . I opened up logs on both the web server and the Gitlab server’s nginx backend, and in the access logs the request was showing 403 on the apache host, but not the Gitlab server. Checking the Apache error logs finally revealed my issue. Modsecurity was flagging the communication between my Gitlab server and its own API. Somehow, the encoding had changed and now resembled a ‘Windows remote command execution’ string, and the corresponding modsecurity rule was blocking the request
Luckily, none of my web hosts run Windows, so all I had to do was disable to two specific Windows RCE rules that were tripping false alarms. To do that I added ‘SecRuleRemoveById ######’ in /etc/apache2/conf-available/security.conf for each of the two rules. Once I restarted Apache, everything was working smoothly again.
Although it’s always annoying to troubleshoot new issues, I definitely learned a lot from fixing this issue. I learned a lot more about different configurations for reverse proxies in Apache, and I got more experience with troubleshooting with server logs. I think next time, I will check logs first rather than stare at my configs and guess what might be wrong haha 🙂