Custom ContentProvider - 404 on document blob, first load [11.9.2]

Hi,

I am experiencing an issue with our custom ContentProvider (after upgrading to 11.9.2) where a document (pdf) is throwing a 404.

The weird thing is that it only does this the very first time it is loaded into the ContentProvider cache. When I refresh it works as expected but after +-5 min (ContentProvider cache duration) the same thing happens.

Sample url: http://localhost/globalassets/documents-repository/xxx/Latest/Latest.pdf

Execution:

The first time executing the request for the file it runs through the LoadContent method and populates the Blob:

protected override IContent LoadContent(ContentReference contentLink, ILanguageSelector languageSelector)
{
   var content = DocumentRepository.Service.Get(contentLink);
    //Other types handled here
	if (content is DocumentFile)
	{
		var documentContent = GetDefaultContent(ContentLoader.Service.Get<ApiDocumentCategory>(content.ParentLink),
			new string[] { ".JPG", ".JPEG", ".GIF", ".PNG" }.Contains(Path.GetExtension((content as DocumentFile).Href), StringComparer.OrdinalIgnoreCase) ? ContentTypeRepository.Load(typeof(ApiDocumentImage)).ID : ContentTypeRepository.Load(typeof(ApiDocument)).ID,
										languageSelector);
	documentContent.ContentGuid = content.ContentGuid;
	documentContent.ContentLink = contentLink;
	documentContent.Name = content.Name;
	(documentContent as MediaData).Changed = DateTime.Now;
	(documentContent as MediaData).StartPublish = DateTime.Now;
	(documentContent as MediaData).RouteSegment = content.Name;
	(documentContent as MediaData).Status = VersionStatus.Published;
	(documentContent as IApiMetadata).Href = (content as DocumentFile).Href;
	(documentContent as ISupportsDataDate).DataDate = (content as DocumentFile).DataDate;
	(documentContent as IApiMetadata).EntityId = (content as DocumentFile).EntityId;
	(documentContent as IVersionable).IsPendingPublish = false;
	try
	{
		var provider = ServiceLocator.Current.GetInstance<IBlobProviderRegistry>().GetProvider(new Uri("epi.fx.blob://apidoc/"));
		var extension = Path.GetExtension((documentContent as IApiMetadata).Href);
		var blob = provider.CreateBlob((documentContent as MediaData).BinaryDataContainer, string.IsNullOrEmpty(extension) ? ".null" : extension) as ApiDocumentBlob;
		blob.Href = (documentContent as IApiMetadata).Href;
		(documentContent as MediaData).BinaryData = blob;
	}
	catch (Exception ex)
	{
		IApiMetadata apiMeta = (documentContent as IApiMetadata);
		var href = apiMeta == null ? "" : apiMeta.Href;
		Logger.Critical(href + ": " + ex.Message, ex);
		throw;
	}
	return documentContent;
}

Directly after this I get an error in the logs:

2018-11-03 18:23:23,268 [26] ERROR EPiServer.Global: Unhandled exception in ASP.NET
System.Web.HttpException (0x80004005): Not Found.
   at EPiServer.Web.BlobHttpHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
System.Web.HttpException (0x80004005): Not Found.
   at EPiServer.Web.BlobHttpHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

When I refresh, the code steps into the OpenRead method of the custom blob successfully and the file loads correctly:

public class ApiDocumentBlob : Blob
{
	internal string Href { get; set; }
	public ApiDocumentBlob(Uri id)
		: base(id)
	{
	}
	public override System.IO.Stream OpenRead()
	{
		var uriBuilder = new UriBuilder(ApiManager.Instance.ApiDocumentUrl);
		uriBuilder.Path = Href;
		var webClient = new WebClient();
		var url = uriBuilder.Uri.ToString();
		return webClient.OpenRead(url);
	}
}

Subsequent refreshes works as well, due to the ContentProvider cache I presume:

//Other SetCacheSettings overrides look the same
protected override void SetCacheSettings(IContent content, CacheSettings cacheSettings)
{
	var fiveMinutes = TimeSpan.FromMinutes(5);
	cacheSettings.AbsoluteExpiration = DateTime.Now.Add(fiveMinutes);
	cacheSettings.SlidingExpiration = Cache.NoSlidingExpiration;
	base.SetCacheSettings(content, cacheSettings);
}

If I wait 5+ minutes for the cache to expire I experience exactly the same symptoms as above.

The fact that it does work eventually makes me think that the segments etc. are setup correctly.

I am busy trying to setup my own BlobHttpHandler to try and see if I can figure out why exactly it is failing.

Any other ideas would be very much appreciated.