src/libraries/System.Text.Json/docs/ReferenceHandling_spec.md
Reference loops: Also referred as circular references, loops occur when a property of a .NET object refers to the object itself, either directly (a -> a) or indirectly (a -> b -> a). They also occur when the element of an array refers to the array itself (arr[0] -> arr). Multiple occurrences of the same reference do not imply a cycle.
Preserve duplicated references: Semantically represent objects and/or arrays that have been previously written, with a reference to them when found again in the object graph (using reference equality for comparison).
Metadata: Extra properties on JSON objects and/or arrays (that may change their schema) to enable reference preservation when round-tripping. These additional properties are only meant to be understood by the JsonSerializer.
Currently, there is no mechanism to avoid infinite loops while serializing .NET object instances that contain cycles nor to preserve references that round-trip when using System.Text.Json. The JsonSerializer throws a JsonException when a loop is found within the object graph.
This is a heavily requested feature since it is considered by many as a very common scenario, especially when serializing POCOs that came from an ORM Framework, such as Entity Framework; even though the JSON specification does not support reference loops by default. Therefore, this will be implemented as an opt-in feature (for both serialization and deserialization).
The current solution to deal with cycles in the object graph while serializing is to rely on MaxDepth and throw a JsonException after it is exceeded. This was done to avoid perf overhead for cycle detection in the common case. The goal is to enable the new opt-in feature with minimal impact to existing performance.
namespace System.Text.Json
{
public partial class JsonSerializerOptions
{
public ReferenceHandling ReferenceHandling { get; set; } = ReferenceHandling.Default;
}
}
namespace System.Text.Json.Serialization
{
/// <summary>
/// This class defines the various ways the <see cref="JsonSerializer"/>
/// can deal with references on Serialization and Deserialization.
/// </summary>
public sealed class ReferenceHandling
{
public static ReferenceHandling Default { get; }
public static ReferenceHandling Preserve { get; }
// TODO: decide if we keep or remove this option.
public static ReferenceHandling Ignore { get; }
}
}
See also the internal implementation details.
Default:
JsonException when MaxDepth is exceeded. This may occur by either a reference loop or by passing a very deep object. This option will not affect the performance of the serializer.JsonPropertyName or be added to the JsonExtensionData overflow dictionary.Preserve:
$id, $values and $ref) properties in order to reference them later by writing a reference to the previously written JSON object or array.Preserve does affect its behavior, as follows: Metadata will be expected (although is not mandatory) and the deserializer will try to understand it.Ignore:
JsonPropertyName or be added to the JsonExtensionData dictionary.For System.Text.Json, the goal is to stick to the same metadata syntax used when preserving references using Newtonsoft.Json and provide a similar usage in JsonSerializerOptions that encompasses the needed options (e.g. provide reference preservation). This way, JSON output produced by Newtonsoft.Json can be deserialized by System.Text.Json and vice versa.
This API is exposing the ReferenceHandling property as a class, to be extensible in the future; and provide built-in static instances of Default and Preserve that are useful to enable the most common behaviors by just setting those in JsonSerializerOptions.ReferenceHandling.
With ReferenceHandling being a class, we can exclude things that, as of now, we are not sure are required and add them later based on customer feedback. For example, the Object and Array granularity of Newtonsoft.Json's PreserveReferencesHandling feature or the ReferenceLoopHandling.Ignore option.
The next table show the combination of Newtonsoft's ReferenceLoopHandling (RLH) and PreserveReferencesHandling (PRH) and how to get its equivalent on System.Text.Json's ReferenceHandling:
| RLH\PRH | None | All | Objects | Arrays |
|---|---|---|---|---|
| Error | Default | future (overlap) | future (overlap) | future (overlap) |
| Ignore | Ignore | future (overlap) | future (overlap) | future (overlap) |
| Serialize | future | Preserve | future | future |
Notes:
MetadataPropertyHandling.ReadAhead for now.Objects and Arrays granularity may apply to both, serialization and deserialization.Newtonsoft.Json, PreserveReferencesHandling takes precedence); see example.class Employee
{
[JsonPropertyName("$id")]
public string Identifier { get; set; }
public Employee Manager { get; set; }
[JsonExtensionData]
public IDictionary<string, object> ExtensionData { get; set; }
}
private const string json =
@"{
""$id"": ""1"",
""Name"": ""Angela"",
""Manager"": {
""$id"": ""2"",
""Name"": ""Bob"",
""Manager"": {
""$ref"": ""2""
}
}
}";
public static void ReadObject()
{
Employee angela = JsonSerializer.Deserialize<Employee>(json);
Console.WriteLine(angela.Identifier); //prints: "1".
Console.WriteLine(angela.Manager.Identifier); //prints: "2".
Console.WriteLine(angela.Manager.Manager.ExtensionData["$ref"]); //prints: "2".
}
Note how you can annotate .Net properties to use properties that are meant for metadata and are added to the JsonExtensionData overflow dictionary, in case there is any, when opting-out of the ReferenceHanding.Preserve feature.
For the next samples, let's assume you have the following class:
class Employee
{
public string Name { get; set; }
public Employee Manager { get; set; }
public List<Employee> Subordinates { get; set; }
}
private Employee bob = new Employee { Name = "Bob" };
private Employee angela = new Employee { Name = "Angela" };
angela.Manager = bob;
bob.Subordinates = new List<Employee>{ angela };
public static void WriteObject()
{
string json = JsonSerializer.Serialize(angela, options);
// Throws JsonException -
// "A possible object cycle was detected which is not supported.
// This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 64."
}
private Employee bob = new Employee { Name = "Bob" };
private Employee angela = new Employee { Name = "Angela" };
angela.Manager = bob;
bob.Subordinates = new List<Employee>{ angela };
On System.Text.Json:
public static void WriteIgnoringReferenceLoops()
{
var options = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Ignore
WriteIndented = true,
};
string json = JsonSerializer.Serialize(angela, options);
Console.Write(json);
}
On Newtonsoft.Json:
public static void WriteIgnoringReferenceLoops()
{
var settings = new JsonSerializerSettings
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore
Formatting = Formatting.Indented
};
string json = JsonConvert.SerializeObject(angela, settings);
Console.Write(json);
}
Output:
{
"Name": "Angela",
"Manager": {
"Name": "Bob",
// Note how subordinates is empty because Angela is being ignored.
"Subordinates": []
}
}
private Employee bob = new Employee { Name = "Bob" };
private Employee angela = new Employee { Name = "Angela" };
angela.Manager = bob;
bob.Subordinates = new List<Employee>{ angela };
On System.Text.Json:
public static void WritePreservingReference()
{
var options = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Preserve
WriteIndented = true,
};
string json = JsonSerializer.Serialize(angela, options);
Console.Write(json);
}
On Newtonsoft.Json:
public static void WritePreservingReference()
{
var settings = new JsonSerializerSettings
{
PreserveReferencesHandling = PreserveReferencesHandling.All
Formatting = Formatting.Indented
};
string json = JsonConvert.SerializeObject(angela, settings);
Console.Write(json);
}
Output:
{
"$id": "1",
"Name": "Angela",
"Manager": {
"$id": "2",
"Name": "Bob",
"Subordinates": {
// Note how the Subordinates' square braces are replaced with curly braces
// in order to include $id and $values properties,
// $values will now hold whatever value was meant for the Subordinates list.
"$id": "3",
"$values": [
{ // Note how this object denotes reference to Angela that was previously serialized.
"$ref": "1"
}
]
}
}
}
private const string json =
@"{
""$id"": ""1"",
""Name"": ""Angela"",
""Manager"": {
""$id"": ""2"",
""Name"": ""Bob"",
""Subordinates"": {
""$id"": ""3"",
""$values"": [
{
""$ref"": ""1""
}
]
}
}
}";
On System.Text.Json:
public static void ReadJsonWithPreservedReferences(){
var options = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Preserve
};
Employee angela = JsonSerializer.Deserialize<Employee>(json, options);
Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}
On Newtonsoft.Json:
public static void ReadJsonWithPreservedReferences(){
var options = new JsonSerializerSettings
{
//Newtonsoft.Json reads metadata by default, just setting the option for illustrative purposes.
MetadataPropertyHanding = MetadataPropertyHandling.Default
};
Employee angela = JsonConvert.DeserializeObject<Employee>(json, settings);
Console.WriteLine(object.ReferenceEquals(angela, angela.Manager.Subordinates[0])); //prints: true.
}
Newtonsoft.Json contains settings that you can enable to deal with such problems.
When using ReferenceLoopHandling.Ignore, other objects that were already seen on the current graph branch will be ignored on serialization.
When using PreserveReferencesHandling.All you are signaling that your resulting JSON will contain metadata properties $ref, $id and $values which are going to act as reference identifiers ($id) and pointers ($ref).
Now, to read back those references, you have to use MetadataPropertyHandling.Default to indicate that metadata is expected in the payload passed to the Deserialize method.
JsonException.Path, JsonSerializerOptions.IgnoreNullValues, JsonPropertyNameAttribute, and Converters).https://dojotoolkit.org/reference-guide/1.10/dojox/json/ref.html
Similar: https://www.npmjs.com/package/json-cyclic
Newtonsoft.Json)$id nor $values metadata, therefore, everything can be referenced.Newtonsoft.Json.https://github.com/WebReflection/flatted
https://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion
@JsonIdentityInfo where you can define a class property that will be used to further represent the object.As a rule of thumb, we throw on all cases where the JSON payload being read contains any metadata that is impossible to create with the JsonSerializer (e.g. it was hand modified). Since System.Text.Json is more strict, it means that certain payloads that Newtonsoft.Json could process, will fail with System.Text.Json. Specific example scenarios where that could happen are described below.
$ref.
$ref is ignored if a regular property is previously found in the object.{
"$id": "1",
"Name": "Angela",
"Manager": {
"Name": "Bob",
"$ref": "1"
}
}
$ref.
{
"$id": "1",
"Name": "Angela",
"Manager":{
"$ref": "1",
"Name": "Angela"
}
}
$ref:
$id is disregarded, and the reference is set.{
"$id": "1",
"Name": "Angela",
"Manager": {
"$id": "2",
"$ref": "1"
}
}
$ref:
{
"$id": "1",
"Name": "Angela",
"Manager": {
"$ref": "1",
"$id": "2"
}
}
null.[
{
"$ref": "1"
},
{
"$id": "1",
"Name": "Angela"
}
]
$id in the same object:
null (if $ref would be "2", it would evaluate to itself).{
"$id": "1",
"$id": "2",
"Name": "Angela",
"Manager": {
"$ref": "1"
}
}
$id is not the first property:
Newtonsoft.Json: Object is not preserved and cannot be referenced, therefore any reference to it would evaluate as null.
S.T.Json: Throw - Object $id is not the first property.
Note: In case we would want to switch, we can handle the $id not being the first property since we store the reference at the moment we spot the $id property, we throw to honor the rule of thumb.
{
"Name": "Angela",
"$id": "1",
"Manager": {
"$ref": "1"
}
}
$id is duplicated (not necessarily nested):
[
{
"$id": "1",
"Name": "Angela"
},
{
"$id": "1",
"Name": "Bob"
}
]
A regular array is [ elem1, elem2 ].
A preserved array is written in the next format { "$id": "1", "$values": [ elem1, elem2 ] }
Preserved array does not contain any metadata:
{}
Preserved array only contains $id:
{
"$id": "1"
}
Preserved array only contains $values:
{
"$values": []
}
Preserved array $values property is null
{
"$id": "1",
"$values": null
}
Preserved array $values property is a primitive value
{
"$id": "1",
"$values": 1
}
Preserved array $values property contains object
{
"$id": "1",
"$values": {}
}
Preserved array contains a property other than $id and $values
{
"$id": "1",
"$values": [1, 2, 3],
"TrailingProperty": "Hello world"
}
$ref Valid under conditions:
$id Valid under conditions:
$values Not valid
$.* Not valid
\u0024.* valid
\u0024id* valid but not considered metadata.
Note: For Dictionary keys on serialize, should we allow serializing keys $id, $ref and $values? If we allow it, then there is a potential round-tripping issue.
Sample of similar issue with DictionaryKeyPolicy:
public static void TestDictionary_Collision()
{
var root = new Dictionary<string, int>();
root["helloWorld"] = 100;
root["HelloWorld"] = 200;
var opts = new JsonSerializerOptions
{
DictionaryKeyPolicy = JsonNamingPolicy.CamelCase
};
string json = JsonSerializer.Serialize(root, opts);
Console.WriteLine(json);
/* Output:
{"helloWorld":100,"helloWorld":200} */
// Round tripping issue
root = JsonSerializer.Deserialize<Dictionary<string, int>>(json);
}
Resolution for above issue: On serialization, when a JSON property name, that is either a dictionary key or a CLR class property, starts with a '$' character, we must write the escaped character "\u0024" instead.
On deserialization, metadata will be digested by using only the raw bytes, so no encoded characters are allowed in metadata; to read JSON properties that start with a '$' you will need to pass it with the escaped '$' (\u0024) or turn the feature off.
$ref Valid under conditions:
$id Valid under conditions:
$values Valid under conditions:
$id.* Not Valid any property other than above metadata will not be valid.
Since these types are created with the help of an internal converter, and they are not parsed until the entire block of JSON finishes; nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive.
With that said, the deserializer will throw when it reads $id on any of these types. When serializing (e.g. writing) those types, however, they are going to be preserved as any other collection type ({ "$id": "1", "$values": [...] }) since those types can still be parsed into a collection type that is supported.
Note: By the same principle, Newtonsoft.Json does not support parsing JSON arrays into immutables as well.
Note 2: When using immutable types and ReferenceHandling.Preserve, you will not be able to generate payloads that are capables of round-tripping.
ImmutableList and ImmutableDictionaryArray<T> and T[]$id for every JSON complex type. However, to reduce bandwidth, structs will not be written with metadata, since it would be meaningless due ReferenceEquals is used when comparing the objects and no backpointer reference would be ever written to an struct.public static void SerializeStructs()
{
EmployeeStruct angela = new EmployeeStruct
{
Name = "Angela"
};
List<EmployeeStruct> employees = new List<EmployeeStruct>
{
angela,
angela
};
var options = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Preserve
};
string json = JsonSerializer.Serialize(employees, options);
Console.WriteLine(json);
}
Output:
```json
{
"$id": "1",
"$values": [
{
"Name": "Angela"
},
{
"Name": "Angela"
}
]
}
$ref within a property that matches to a value type (such as a struct) and ReferenceHandling.Preserve is set.Example:
public static void DeserializeStructs()
{
string json = @"
{
""$id"": ""1"",
""$values"": [
{
""$id"": ""2"",
""Name"": ""Angela""
},
{
""$ref"": ""2""
}
]
}";
var options = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Preserve
};
List<EmployeeStruct> root = JsonSerializer.Deserialize<List<EmployeeStruct>>(json, options);
// Throws JsonException.
}
In other words, having a $ref property in a struct, is never emitted by the serializer and reading such a payload (for instance, if the payload was hand-crafted) is not supported by the deserializer. However, since Newtonsoft.Json does emit $id for value-type objects System.Text.Json will allow reading struct objects that contain $id, regardless of not being able to create such payloads.
Let's say you have the following class:
private class EmployeeAnnotated
{
[JsonPropertyName("$id")]
public string Identifier { get; set; }
[JsonPropertyName("$ref")]
public string Reference { get; set; }
[JsonPropertyName("$values")]
public List<EmployeeAnnotated> Values { get; set; }
public string Name { get; set; }
}
Both on serialization and deserialization:
public static void DeSerializeWithPreserve()
{
var root = new EmployeeAnnotated();
var opts = new JsonSerializerOptions
{
ReferenceHandling = ReferenceHandling.Preserve
};
// The property will be emitted with the '$' encoded.
string json = JsonSerializer.Serialize(root, opts);
Console.WriteLine(json);
}
{
"\u0024id": null,
"\u0024ref": null,
"\u0024values": null
}
If the name of your property starts with '$', either by using JsonPropertyNameAttribute, by using F#, or by any other reason, that leading '$' (and that one only), will be replaced with its encoded equivalent \u0024.
Things that we may want to consider building on top based on customer feedback:
(De)Serialize can define its own, independently configurable reference handling behavior (for example, you could opt-out from preserve reference on serialization but opt-in for reading them on deserialization).
Expose a ReferenceResolver to override the logic that preserves references (the caller creates their own implementation of a reference resolver).
Expose the ReferenceResolver in Converters to have access to the map of references.
Create JsonReferenceHandlingAttribute to enable annotating properties and classes with their own isolated ReferenceHandling behavior (I am deferring this feature because the constant checking for attributes was causing too much perf overhead on the main path, but maybe we can try moving the attribute check to the warm-up method to reduce the runtime increase).
// Example of a class annotated with JsonReferenceHandling attributes.
[JsonReferenceHandling(ReferenceHandling.Preserve)]
public class Employee {
public string Name { get; set; }
[JsonReferenceHandling(ReferenceHandling.Ignore)]
public Employee Manager { get; set; }
public List<Employee> Subordinates { get; set; }
}
ReferenceHandling (shows potential API evolution and usage).public static void WriteIgnoringReferenceLoopsAndReadPreservedReferences()
{
var bob = new Employee { Name = "Bob" };
var angela = new Employee { Name = "Angela" };
angela.Manager = bob;
bob.Subordinates = new List<Employee>{ angela };
var allEmployees = new List<Employee>
{
angela,
bob
};
var options = new JsonSerializerOptions
{
ReferenceHandling = new ReferenceHandling(
PreserveReferencesHandling.All, // Preserve References Handling on serialization.
PreserveReferencesHandling.All, // Preserve References Handling on deserialization.
ReferenceLoopHandling.Ignore) // Reference Loop Handling on serialization.
WriteIndented = true,
};
string json = JsonSerializer.Serialize(allEmployees, options);
Console.Write(json);
/* Output:
[
{
"$id": "1",
"Name": "Angela",
"Manager": {
"$id": "2",
"Name": "Bob",
"Subordinates": {
"$id": "3",
// Note how subordinates is empty because Angela is being ignored.
// Alternatively: we may let PreserveReferenceHandling take precedence and write the reference instead?
"$values": []
}
}
},
{
// Note how element 2 is written as a reference
// since was previously seen in allEmployees[0].Manager
"$ref": "2"
}
]
*/
allEmployees = JsonSerializer.Deserialize<List<Employee>>(json, options);
Console.WriteLine(allEmployees[0].Manager == allEmployees[1]);
/* Output: true */
}
MaxDepth validation will not be affected by ReferenceHandling.Preserve.Newtonsoft.Json types ReferenceLoopHandling, MetadataPropertyHandling (without ReadAhead), and PreserveReferencesHandling (without the granularity of Objects and Arrays) into one single class; ReferenceHandling.System.Arrays can be serialized with preserve semantics, they will not be supported when trying to deserialize them as a reference. Those types are created with the help of an internal converter and they are not parsed until the entire block of JSON finishes. Nested reference to these types is impossible to identify, unless you re-scan the resulting object, which is too expensive.ReferenceResolver, JsonPropertyAttribute.IsReference and JsonPropertyAttribute.ReferenceLoopHandling, that build on top of ReferenceLoopHandling and PreserveReferencesHandling were considered but they can be added in the future based on customer requests.ReferenceHandling.Ignore. This option will not ship if said evidence is not found.JsonExtensionData is currently not supported (we emit the metadata on serialization and we create a JsonElement on deserialization instead), while in Newtonsoft.Json they are supported. This may change in a future based on customer feedback.