Codable and missing keys

@abrownsword

Mar '20

I'm finding that Codable and JSONEncoder/Decoder are frustratingly close to what I need, but are missing one feature that would enable my (fairly simple!) use case... and I haven't figured out how to add this feature myself.

There are a thousand articles to be found on how to make your types Codable and then serialize/deserialize to JSON. I've implemented this, and it works great with minimal effort and code. The problem showed up when I added a field to my type. The JSONDecoder then immediately refused to deserialize any existing data, and it throws an error without having done any of the deserialization work. That simply isn't acceptable.

The solutions I have found involve either:

Make all my fields optional. This is not acceptable because the fields AREN'T optional, they must be there. But they have perfectly reasonable defaults. And making them optional results in tremendous code pollution.
Implement fully manual encoder/decoder methods across all my types and maintain forever more. This is what this whole scheme was supposed to do away with as it results in a lot of manually written code and potential bugs... and the compiler is already auto-implementing these for me!!
Implement a parallel set of types to serialize into and then copy that data into my actual types. Sorry, this is just stupid -- double the number of types to maintain, not to mention the copy functions.
Implement some sort of generic wrapper on each field of each type, which isn't much better than #1 and causes complications if there are property wrappers or other things going on.

There are 3 very simple situations that happen very frequently when types are being modified:

Field is added, in which case old serialized state is missing the field and its key -- JSONDecoder will barf when I just want it to ignore skip the field, leaving it as its default value.
Field is removed, in which case the old serialized state is data that will not be used. JSONDecoder doesn't appear to have this issue.
Field is renamed, in which case the coding key remapping could be used (at dev's discretion).

With some careful thought/planning when making changes to the types, this simple strategy can go a LONG way to staying compatible with existing saved data, and it involves much much less code that any of the solutions (1..4) that I've found so far. It is thus very frustrating that it doesn't seem possible to tell JSONDecoder (or the compiler's decoder method generator) to simply ignore missing keys and carry on.

I can think of three potential solutions, but don't know if any of them are actually possible...

Is there an option in the compiler to ignore missing fields when generating the decoder function? (unlikely as I haven't seen it mentioned)
Is there a way to provide per-field error handling that can skip instead of throw? (also unlikely)
Is it possible to implement a Decoder protocol implementation that somehow wraps JSONDecoder? (doesn't seem straightforward as JSONDecoder apparently doesn't implement Decoder itself!)

Anyone have a workable solution?

@PeterKaminski09

Mar '20

Make all my fields optional. This is not acceptable because the fields AREN'T optional, they must be there. But they have perfectly reasonable defaults. And making them optional results in tremendous code pollution.

One possible way to reframe your problem is to consider separating the domain model from the data model. You could implement a DTO for the data model that conforms to Codable and has optional fields that are populated by the Decoder. Then you could have a domain model and some mapper that hydrates the domain model from the DTO/data model. The domain model could be responsible for providing these reasonable default values that you mentioned.

It also has the added benefit of abstracting the rest of your code from how your data model/persistence layer works under the hood, and you could easily change persistence layers without too much overhead.

Here's a pretty good article that sums of the pros of doing this

https://medium.com/better-programming/why-model-objects-shouldnt-implement-swift-s-decodable-or-encodable-protocols-1249cb44d4b3

I do like your idea of some sort of option for the decoder to skip missing keys, that seems pretty sensible.

@abrownsword

Mar '20

Yes, I know I could implement a DTO layer (it was solution #3 in my first list)... but it is a lot of code to maintain. My use case is to have a simple and easy to maintain serialization module. Duplicating or contorting all the types and providing helper functions goes directly against the principle of it. I fully understand the use cases where DTO models and the other solutions make sense (and I've done pretty much all the variations at one time or another... have been at this game for a loooong time), but sometimes just being able to serialize the data and having a less strict deserialization is really just the best solution. Unfortunately in Swift it appears this isn't currently possible, which is too bad because they are just a whisker away from one of the best solutions I've ever seen to this problem.

I had been hoping to write my own wrapper around JSONDecoder, but my Swift Fu is weak and I'm not sure I could pull it off without a major investment in time that I don't really want to make currently.

Sponsor Hacking with Swift and reach the world's largest Swift community!

	@abrownsword Mar '20 I just stumbled upon this, which looks promising: https://github.com/marksands/BetterCodable Will need to verify that it does what I need and that all the property wrapping doesnt mess with the other builtin compiler behaviours I'm using (e.g. Equatable). 3
	@guseulalio Apr '20 Sounds like a file type version problem. Maybe try this. Add a `version` field to top level of your model. Before you read the model, read only the `version` field. The problem here, is you'll have manage the versions in code too, if you still want to read the old files. Otherwise, just check if the version is the latest one before decoding the whole file. 3
	@abrownsword Apr '20 Yeah, and forcing the versioning into the code is far more work than the problem deserves. Which is why I want a simple decode policy that can tolerate changes in the format. 4
	@PeterKaminski09 Apr '20 Saw this on the Swift forums summary email, may be worth a look: https://forums.swift.org/t/introducing-resilient-decoding/35212 3
	@abrownsword May '20 Thanks. It doesn't do what I want (i.e. support simple value types that are missing), but I have posted an issue to the Github project... maybe he can add that functionality. 3
	@MaverickSwap Jul '20 It's really annoying that we cannot easily setup default values to variables when using Decodable. Even though there are various other ways to implement an init just to make it work, it defies it's purpose of keeping it small & tidy! I'm working with an unpredictable response which might give a nil for any key and I need to initialise the model properties with various default values if it's not in the response. I find the traditional approach of parsing the dictionary into model makes more sense if you want a model with many default values. It certainly adds a bit of a boilerplate but definitely gives a finer control over default values. You can try out this - https://mavswap-swift-model-generator.herokuapp.com/ It takes in a json response and returns the models along with dictionary parsing. 3
	@lbarbosa Jul '20 I share your frustration and could identify not only with the issue but also with the issues of the most common solutions. Anyway, I ran into a couple of articles that discussed using a `@propertyWrapper` to overcome such common issues. Although it still has some limitations, for the majority of cases where you just want to assign a sane default when a key is missing, it's great. Particularly when dealing with an established codebase with objects with dozens of properties. You can find more about this on my answer to a similar stackoverflow question: https://stackoverflow.com/a/63103704/2030092 Or jump straight to the article/library that implements some useful examples: http://marksands.github.io/2019/10/21/better-codable-through-property-wrappers.html https://github.com/marksands/BetterCodable 3

Sponsor Hacking with Swift and reach the world's largest Swift community!

Archived topic

This topic has been closed due to inactivity, so you can't reply. Please create a new topic if you need to.

All interactions here are governed by our code of conduct.