ContextMode inside filter

We have a filter that configures NWebSec security headers on the response and is to be applied only when not in edit or preview. Trying to find a good way of controlling wether the request is from edit/preview or not.

The only working solution I have at the moment is to pickup the uiUrl parameter from the configuration and check if that part exists in the current Request, like this:

 var uiUrl = EPiServer.Configuration.Settings.Instance.UIUrl.ToString().Replace("~", string.Empty).ToLower().Split('/')[0];
                var currentUrl = HttpContext.Current.Request.Url.GetLeftPart(UriPartial.Path).ToLower();
                isInEditMode = currentUrl.Contains($"/{uiUrl}/");

I also tried the ContextModeResolver inside my FilterProvider GetFilters method, like this:

controllerContext.RequestContext.GetContextMode().EditOrPreview();

But it turns out that all requests that EPiserver does inside the edit mode that has to do with setting up the UI all have the Default mode and returns false to the check above. Only the request to load the specific preview of the page returns true. This makes the filter add headers to all Episerver requests thus making the EditUI to fail and not load correct.

Is there a better way to apply this filter to requests outside of the EditUI than to check the current url for UIUrl key?