Broken DnD with backingtype PropertyPageReference but property type is custom content

I have this:

        [BackingType(typeof(PropertyContentReference))]
        [AllowedTypes(typeof(Contact)), Required]
        public virtual Contact Contact
        {
            get
            {
                var contentRef = this.GetPropertyValue<ContentReference>(this.GetPropertyName(x => x.Contact));
                if (ContentReference.IsNullOrEmpty(contentRef))
                    return null;
                return ServiceLocator.Current.GetInstance<IContentLoader>().Get<Contact>(contentRef);
            }
            set
            {
                this.SetValue(this.GetPropertyName(x => x.Contact), value);
            }
        }

The property works as I like, but DnD is borken. I get an error in the UI: