JSON deserialization in Swift

Image from: https://unsplash.com/photos/fvddO05Z20Y
Author of this article: Wufan

Several schemes commonly used in the industry

Manual decoding schemes such as Unbox ( DEPRECATED )

The scheme commonly used in Swift's early days, similar to ObjectMapper

This solution requires users to manually write decoding logic, and the cost of use is relatively high; it has been replaced by Codable officially launched by Swift.

Example:

 struct User {
    let name: String
    let age: Int
}

extension User: Unboxable {
    init(unboxer: Unboxer) throws {
        self.name = try unboxer.unbox(key: "name")
        self.age = try unboxer.unbox(key: "age")
    }
}

Ali's open source HandyJSON

HandyJSON currently relies on memory rules inferred from the Swift Runtime source code to operate directly on memory.

In terms of use, there is no need for complicated definitions, no need to inherit from NSObject, and it is enough to declare that the protocol is implemented.

Example:

 class Model: HandyJSON {
    var userId: String = ""
    var nickname: String = ""
    
    required init() {}
}

let jsonObject: [String: Any] = [
    "userId": "1234",
    "nickname": "lilei",
] 

let model = Model.deserialize(from: object)

However, there are compatibility and security issues. Due to the strong reliance on memory layout rules, there may be stability issues when upgrading a major version of Swift. At the same time, because the data structure needs to be parsed through reflection at runtime, it will have a certain impact on performance.

Sourcery -based metaprogramming solution

Sourcery is a Swift code generator that uses SourceKitten to parse Swift source code and generate final code based on Stencil templates

The ability to customize is very strong, which can basically meet all our needs

Example:

Define the AutoCodable protocol, and let the data types that need to be parsed follow this protocol

 protocol AutoCodable: Codable {}

class Model: AutoCodable {
    // sourcery: key = "userID"
    var userId: String = ""
    var nickname: String = ""
    
    required init(from decoder: Decoder) throws {
        try autoDecodeModel(from: decoder)
    }
}

Then generate code through Sourcery . In this process, Sourcery will scan all codes and automatically generate parsing code for classes/structures that implement the AutoCodable protocol.

 // AutoCodable.generated.swift
// MARK: - Model Codable
extension Model {
    enum CodingKeys: String, CodingKey {
        case userId = "userID"
        case nickname
    }

    // sourcery:inline:Model.AutoCodable
    public func autoDecodeModel(from decoder: Decoder) throws {
        // ...
    }
}

As shown above, you can also implement custom functions such as key-value mapping through code comments (annotations), but you need to have strong specification requirements for users. Secondly, in the process of componentization, each component needs to be invaded/renovated, which can be solved by the internal team through the tool chain, which may not be very suitable as a cross-team general solution

Swift build-in API Codable

The JSON serialization scheme officially launched after Swift 4.0 can be understood as a combination of Unbox+Sourcery. The compiler will automatically generate the codec logic according to the data structure definition, and the developer will use a specific Decoder/Encoder to transform the data.

Codable is an official solution launched by Swift, and users can access it at no cost. However, in the process of specific practice, some problems were encountered

• Key value mapping is not friendly, such as the following cases:

 // swift
struct User: Codable {
    var name: String
    var age: Int
    // ...
}

// json1
{
    "name": "lilei"
}

// json2
{
    "nickname": "lilei"
}

// json3
{
    "nickName": "lilei"
}

The Swift compiler will automatically generate complete CodingKeys for us, but if you need to parse the nickname or nickName in json into User.name , you need to rewrite the entire CodingKeys, including other irrelevant attributes such as age

• Insufficient fault tolerance and inability to provide default values

  One of the original intentions of Swift's design is security, so it is reasonable for some types of strong verification from a design point of view, but it will increase some usage costs for actual users.

  for example:

 enum City: String, Codable {
    case beijing
    case shanghai
    case hangzhou
}

struct User: Codable {
    var name: String
    var city: City?
}

// json1
{
    "name": "lilei",
    "city": "hangzhou"
}

// json2
{
    "name": "lilei"
}

// json3
{
    "name": "lilei",
    "city": "shenzhen"
}

let decoder = JSONDecoder()

try {
    let user = try? decoder.decode(User.self, data: jsonData3)
}
catch {
    // json3 格式会进入该分支
    print("decode user error")
}

 上述代码中，json1 和 json2 可以正确反序列化成 User 结构，json3 由于 “shenzhen” 无法转化成 City，导致整个 User 结构解析失败，而不是 name 解析成功，city 失败后变成 nil

• Nested structure parsing is cumbersome
• JSONDecoder only accepts data and does not support dict. There is a performance loss in type conversion when used in special scenarios.

Property decorators like BetterCodable

The new language features of Swift 5.0 can be used to supplement some of the original Codable solutions, such as supporting default values, custom parsing strategies, etc. The specific principles are relatively simple, and those who are interested can learn about them by themselves.

Example:

 struct UserPrivilege: Codable {
    @DefaultFalse var isAdmin: Bool
}

let json = #"{ "isAdmin": null }"#.data(using: .utf8)!
let result = try JSONDecoder().decode(Response.self, from: json)

print(result) // UserPrivilege(isAdmin: false)

However, in actual coding, it is necessary to explicitly describe the attributes of the data structure, which increases the cost of use.

Comparison of advantages and disadvantages of each program

	Codable	HandyJSON	BetterCodable	Sourcery
Type compatible	❌	✅	✅	✅
Default value is supported	❌	✅	✅	✅
key-value map	❌	✅	❌	✅
access/usage cost	✅	✅	❌	❌
safety	✅	❌	✅	✅
performance	✅	❌	✅	✅

The above solutions have their own advantages and disadvantages. Based on this, we hope to find a more suitable solution for cloud music. In terms of access and cost of use, Codable is undoubtedly the best choice. The key point is how to solve the existing problems.

Introduction to Codable

Principle Analysis

First look at a set of data structure definitions, the data structure follows the Codable protocol

 enum Gender: Int, Codable {
    case unknown
    case male
    case female
}

struct User: Codable {
    var name: String
    var age: Int
    var gender: Gender
}

Use the command swiftc main.swift -emit-sil | xcrun swift-demangle > main.sil to generate SIL ( Swift Intermediate Language ), and analyze what the compiler does

You can see that the compiler will automatically generate CodingKeys enumeration and init(from decoder: Decoder) throws methods for us

 enum Gender : Int, Decodable & Encodable {
  case unknown
  case male
  case female
  init?(rawValue: Int)
  typealias RawValue = Int
  var rawValue: Int { get }
}

struct User : Decodable & Encodable {
  @_hasStorage var name: String { get set }
  @_hasStorage var age: Int { get set }
  @_hasStorage var gender: Gender { get set }
  enum CodingKeys : CodingKey {
    case name
    case age
    case gender
    @_implements(Equatable, ==(_:_:)) static func __derived_enum_equals(_ a: User.CodingKeys, _ b: User.CodingKeys) -> Bool
    func hash(into hasher: inout Hasher)
    init?(stringValue: String)
    init?(intValue: Int)
    var hashValue: Int { get }
    var intValue: Int? { get }
    var stringValue: String { get }
  }
  func encode(to encoder: Encoder) throws
  init(from decoder: Decoder) throws
  init(name: String, age: Int, gender: Gender)
}

The following is an excerpt of some of the SIL fragments used for decoding. Unfamiliar readers can skip this part and look directly at the pseudocode translated later.

 // User.init(from:)
sil hidden [ossa] @$s6source4UserV4fromACs7Decoder_p_tKcfC : $@convention(method) (@in Decoder, @thin User.Type) -> (@owned User, @error Error) {
// %0 "decoder"                                   // users: %83, %60, %8, %5
// %1 "$metatype"
bb0(%0 : $*Decoder, %1 : $@thin User.Type):
  %2 = alloc_box ${ var User }, var, name "self"  // user: %3
  %3 = mark_uninitialized [rootself] %2 : ${ var User } // users: %84, %61, %4
  %4 = project_box %3 : ${ var User }, 0          // users: %59, %52, %36, %23
  debug_value %0 : $*Decoder, let, name "decoder", argno 1, implicit, expr op_deref // id: %5
  debug_value undef : $Error, var, name "$error", argno 2 // id: %6
  %7 = alloc_stack [lexical] $KeyedDecodingContainer<User.CodingKeys>, let, name "container", implicit // users: %58, %57, %48, %80, %79, %33, %74, %73, %20, %69, %68, %12, %64
  %8 = open_existential_addr immutable_access %0 : $*Decoder to $*@opened("6CB1A110-E4DA-11EC-8A4C-8A05F3D75FB2") Decoder // users: %12, %12, %11
  %9 = metatype $@thin User.CodingKeys.Type
  %10 = metatype $@thick User.CodingKeys.Type     // user: %12
  %11 = witness_method $@opened("6CB1A110-E4DA-11EC-8A4C-8A05F3D75FB2") Decoder, #Decoder.container : <Self where Self : Decoder><Key where Key : CodingKey> (Self) -> (Key.Type) throws -> KeyedDecodingContainer<Key>, %8 : $*@opened("6CB1A110-E4DA-11EC-8A4C-8A05F3D75FB2") Decoder : $@convention(witness_method: Decoder) <τ_0_0 where τ_0_0 : Decoder><τ_1_0 where τ_1_0 : CodingKey> (@thick τ_1_0.Type, @in_guaranteed τ_0_0) -> (@out KeyedDecodingContainer<τ_1_0>, @error Error) // type-defs: %8; user: %12
  try_apply %11<@opened("6CB1A110-E4DA-11EC-8A4C-8A05F3D75FB2") Decoder, User.CodingKeys>(%7, %10, %8) : $@convention(witness_method: Decoder) <τ_0_0 where τ_0_0 : Decoder><τ_1_0 where τ_1_0 : CodingKey> (@thick τ_1_0.Type, @in_guaranteed τ_0_0) -> (@out KeyedDecodingContainer<τ_1_0>, @error Error), normal bb1, error bb5 // type-defs: %8; id: %12

bb1(%13 : $()):                                   // Preds: bb0
  %14 = metatype $@thin String.Type               // user: %20
  %15 = metatype $@thin User.CodingKeys.Type
  %16 = enum $User.CodingKeys, #User.CodingKeys.name!enumelt // user: %18
  %17 = alloc_stack $User.CodingKeys              // users: %22, %20, %67, %18
  store %16 to [trivial] %17 : $*User.CodingKeys  // id: %18
  // function_ref KeyedDecodingContainer.decode(_:forKey:)
  %19 = function_ref @$ss22KeyedDecodingContainerV6decode_6forKeyS2Sm_xtKF : $@convention(method) <τ_0_0 where τ_0_0 : CodingKey> (@thin String.Type, @in_guaranteed τ_0_0, @in_guaranteed KeyedDecodingContainer<τ_0_0>) -> (@owned String, @error Error) // user: %20
  try_apply %19<User.CodingKeys>(%14, %17, %7) : $@convention(method) <τ_0_0 where τ_0_0 : CodingKey> (@thin String.Type, @in_guaranteed τ_0_0, @in_guaranteed KeyedDecodingContainer<τ_0_0>) -> (@owned String, @error Error), normal bb2, error bb6 // id: %20

// %21                                            // user: %25
bb2(%21 : @owned $String):                        // Preds: bb1
  dealloc_stack %17 : $*User.CodingKeys           // id: %22
  %23 = begin_access [modify] [unknown] %4 : $*User // users: %26, %24
  %24 = struct_element_addr %23 : $*User, #User.name // user: %25
  assign %21 to %24 : $*String                    // id: %25
  end_access %23 : $*User                         // id: %26
  ...

Roughly, the container is obtained from the decoder, the specific value is parsed through the decode method, and the corresponding Swift code is translated as follows:

 init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: User.CodingKeys.Type)
    self.name = try container.decode(String.self, forKey: .name)
    self.age = try container.decode(Int.self, forKey: .age)
    self.gender = try container.decode(Gender.self, forKey: .gender)
}

It can be seen that the key part of deserialization is on Decoder , and the commonly used JSONDecoder is an implementation of the Decoder protocol

We cannot manually intervene in the code automatically generated by the compiler. If we want the deserialization result to meet our expectations, we need to customize a Decoder.

The Swift standard library part is open source, and interested students can move to it JSONDecoder.swift

Decoder, Container Protocol

 public protocol Decoder {
    var codingPath: [CodingKey] { get }
    var userInfo: [CodingUserInfoKey : Any] { get }
    func container<Key>(keyedBy type: Key.Type) throws -> KeyedDecodingContainer<Key> where Key : CodingKey
    func unkeyedContainer() throws -> UnkeyedDecodingContainer
    func singleValueContainer() throws -> SingleValueDecodingContainer
}

Decoder contains 3 types of containers, the specific relationships are as follows

Containers need to implement their own decode methods for specific analysis work

KeyedDecodingContainerProtocol - key-value dictionary container protocol ( KeyedDecodingContainer is used for type erasure)

 func decodeNil(forKey key: Self.Key) throws -> Bool
func decode(_ type: Bool.Type, forKey key: Self.Key) throws -> Bool
func decode(_ type: String.Type, forKey key: Self.Key) throws -> String
...
func decodeIfPresent(_ type: Bool.Type, forKey key: Self.Key) throws -> Bool?
func decodeIfPresent(_ type: String.Type, forKey key: Self.Key) throws -> String?
...

SingleValueDecodingContainer - Single Value Container Protocol

 func decode(_ type: UInt8.Type) throws -> UInt8
...
func decode<T>(_ type: T.Type) throws -> T where T : Decodable

UnkeyedDecodingContainer - Array container protocol

 mutating func decodeNil() throws -> Bool
mutating func decode(_ type: Int64.Type) throws -> Int64
mutating func decode(_ type: String.Type) throws -> String
...
mutating func decodeIfPresent(_ type: Bool.Type) throws -> Bool?
mutating func decodeIfPresent(_ type: String.Type) throws -> String?

Typical JSONDecoder usage posture

 let data = ...
let decoder = JSONDecoder()
let user = try? decoder.decode(User.self, from: data)

The analysis process is as follows:

The core parsing logic of Decoder is inside the Container. The following part of the logic will be designed and implemented according to our needs

Self-developed program

feature design

First of all, we need to clarify the effect we need in the end

1. Default value is supported
2. The types are compatible with each other, for example, the int type in JSON can be correctly parsed as the String type in the Model
3. Decoding failure is allowed to return nil instead of directly determining that the decoding process failed
4. Support key mapping
5. Support custom decoding logic

The following protocols are defined here

• Default value protocol, default values of common types are implemented by default, and custom types can also be implemented on demand

 public protocol NECodableDefaultValue {
    static func codableDefaultValue() -> Self
}

extension Bool: NECodableDefaultValue {
    public static func codableDefaultValue() -> Self { false }
}
extension Int: NECodableDefaultValue {
    public static func codableDefaultValue() -> Self { 0 }
}
...

• key value mapping protocol

 public protocol NECodableMapperValue {
    var mappingKeys: [String] { get }
}

extension String: NECodableMapperValue {
    public var mappingKeys: [String] {
        return [self]
    }
}

extension Array: NECodableMapperValue where Element == String {
    public var mappingKeys: [String] {
        return self
    }
}

• Codable protocol extension

 public protocol NECodable: Codable {
    // key 值映射关系定义，类似 YYModel 功能
    static var modelCustomPropertyMapper: [String: NECodableMapperValue]? { get }
    
    // 除了 NECodableDefaultValue 返回的默认值，还可以在该函数中定义默认值
    static func decodingDefaultValue<CodingKeys: CodingKey>(for key: CodingKeys) -> Any?

    // 在解析完数据结构之后，提供二次修改的机会
    mutating func decodingCustomTransform(from jsonObject: Any, decoder: Decoder) throws -> Bool
}

• final use position

 struct Model: NECodable {
    var nickName: String
    var age: Int
    
    static var modelCustomPropertyMapper: [String : NECodableMapperValue]? = [
        "nickName": ["nickname", "nickName"],
        "age": "userInfo.age"
    ]

    static func decodingDefaultValue<CodingKeys>(for key: CodingKeys) -> Any? where CodingKeys : CodingKey {
        guard let key = key as? Self.CodingKeys else { return nil }
        switch key {
        case .age:
            // 提供默认年龄
            return 18
        default:
            return nil
        }
    }
}

let jsonObject: [String: Any] = [
    "nickname": "lilei",
    "userInfo": [
        "age": 123
    ],
]

let model = try NEJSONDecoder().decode(Model.self, jsonObject: jsonObject)

XCTAssert(model.nickName == "lilei")
XCTAssert(model.age == 123)

Decoder, Container specific implementation

Define the class NEJSONDecoder as the specific implementation of the Decoder protocol, and also implement three container protocols

A large number of decode methods need to be implemented inside the container to parse specific values. We can abstract a tool class to perform corresponding type analysis, conversion, and provide default values.

A part of the keyedContainer implementation is given below. The general process is as follows:

1. The entry method called first, which gets the original value from JSON according to key and keyMapping
2. Through the unbox method, convert the original value (possibly String, Int type) into the expected type (such as Bool)

3. If the above process fails, enter the default value processing flow

   1. First get the default value through the decodingDefaultValue method defined by the model, if not, go to step b
   2. Get the default value of the type through the NECodableDefaultValue protocol
4. parsing complete

 class NEJSONKeyedDecodingContainer<K : CodingKey> : KeyedDecodingContainerProtocol {
        public func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool {
        do {
            return try _decode(type, forKey: key)
        }
        catch {
            if let value = self.defaultValue(for: key),
               let unbox = try? decoder.unbox(value, as: Bool.self) { return unbox }
            
            if self.provideDefaultValue {
                return Bool.codableDefaultValue()
            }
            throw error
        }
    }

        public func _decode(_ type: Bool.Type, forKey key: Key) throws -> Bool {
        guard let entry = self.entry(for: key) else {
            throw ...
        }

        self.decoder.codingPath.append(key)
        defer { self.decoder.codingPath.removeLast() }

        guard let value = try self.decoder.unbox(entry, as: Bool.self) else {
            throw ...
        }

        return value
    }
}

Revisit PropertyWrapper

In the NECodable protocol, the usage habit of YYModel is retained, key mapping and default value provide two methods that need to implement the NECodable protocol separately

Using Swift's property decorator, developers can more easily implement the above functions:

 @propertyWrapper
class NECodingValue<Value: Codable>: Codable {
    public convenience init(wrappedValue: Value) {
        self.init(storageValue: wrappedValue, keys: nil)
    }
    
    public convenience init(wrappedValue: Value, keys: String...) {
        self.init(storageValue: wrappedValue, keys: keys)
    }
    
    public convenience init<T>(wrappedValue: Optional<T> = .none, keys: String...) where Value == Optional<T> {
        self.init(storageValue: wrappedValue, keys: [])
    }
    
    public convenience init(keys: String...) {
        self.init(keys: keys)
    }

    // ....
}

struct Model: NECodable {
    @NECodingValue(keys: "nickname")
    var name: String

    // JSON 中不存在时，默认为 hangzhou
    @NECodingValue
    var city: String = "hangzhou"

    // JSON 中不存在时，默认为 false
    var enable: Bool
}

The implementation method is more tricky:

Wrap the instance variable through the attribute decorator, NECodingValue(keys: "nickname") the instance is first initialized, which contains the keys and wrapperValue we defined, and then the init(from decoder: Decoder) process is generated through the decoder NECodingValue(from: decoder) variable is assigned to the _name attribute, at this time the first NECodingValue variable will be released, thus obtaining a code execution time for the customized decoding process (the defaultValue Copy over, decode with custom key, etc...)

Application Scenario Example

Deserialization is usually used to process the data returned by the server. Based on Swift's grammatical features, we can define a network request protocol very simply, for example:

network request protocol

 protocol APIRequest {
    associatedtype Model

    var path: String { get }
    var parameters: [String: Any]? { get }
    
    static func parse(_ data: Any) throws -> Model
}

// 缺省实现
extension APIRequest {
    var parameters: [String: Any]? { nil }

    static func parse(_ data: Any) throws -> Model {
        throw APIError.dataExceptionError()
    }
}

Extend APIRequest protocol, automatically deserialize through Swift's type matching pattern

 extension APIRequest where Model: NECodable {
    static func parse(_ data: Any) throws -> Model {
        let decoder = NEJSONDecoder()
        return try decoder.decode(Model.self, jsonObject: data)
    }
}

Extension APIRequest protocol, add network request method

 extension APIRequest {
    @discardableResult
    func start(completion: @escaping (Result<Model, APIError>) -> Void) -> APIToken<Self> {
        // 具体的网络请求流程，基于底层网络库实现
    }
}

Finally, the business side can simply define a network interface and initiate a request

 // 网络接口定义
struct MainRequest: APIRequest {
    struct Model: NECodable {
        struct Item: NECodable {
            var title: String
        }
        var items: [Item]
        var page: Int
    }

    let path = "/api/main"
}

// 业务侧发起网络请求
func doRequest() {
    MainRequest().start { result in
        switch result {
            case .success(let model):
                // to do something
                print("page index: \(model.page)")
            case .failure(let error):
                HUD.show(error: error)
        }
    }
}

unit test

There are many edge cases in the serialization/deserialization process, and unit tests need to be constructed for each scenario to ensure that all behaviors meet expectations

Performance comparison

The above picture is the result obtained after each deserialization library is executed 10,000 times. It may be seen that JSONDecoder has the best performance when converting from Data to Model, NEJSONDecoder has the best performance when converting from JSON Object to Model, and HandyJSON takes the longest time.

Test code: test.swift

This article is published from the NetEase Cloud Music technical team, and any form of reprinting of the article is prohibited without authorization. We recruit various technical positions all year round. If you are ready to change jobs and happen to like cloud music, then join us at grp.music-fe(at)corp.netease.com!