Using a SelectionFactory with ConentReference property issue

I am attempting to use a selection factory to make choosing a specific kind of content type simpler. I am using a variation of this type of selection factory to list my content in a dropdown:
https://github.com/alfnilsson/episerverutils/blob/master/SelectContent/ContentSelectionFactory.cs

The problem is the property I am using the factory on will not serialize the selected value into a ContentReference properly. I get an error "Unable to cast object of type 'System.String' to type 'EPiServer.Core.ContentReference'" when saving the item. It works fine if I change the property to a string though, but it forces me to parse it into a ContentReference manually if I want to use it in code. Here is my property definition:

[Display(
GroupName = Global.TabNames.Location,
Name = "Location",
Order = 20)]
[SelectOne(SelectionFactoryType = typeof(LocationSelectionFactory))]
public virtual ContentReference Location { get; set; }

My selection factory uses this code to create the select items:

foreach (var location in locations) {
list.Add(new SelectItem {Text = location.Name, Value = location.ContentLink.ToReferenceWithoutVersion()});
}

Is there a step somewhere I am missing that can help with the conversion so I don't have to revert to using a string field?