The App is loading data from a json during the first launch. That data is inserted in SwiftData and synced to the cloud. So far so good. Now the issue is when the App is launched on a different device with the same account the json is loaded and obviously the data is duplicated. In my model an id (UUID) is loaded from the json, so duplicates have the same IDs. I can't use @Attribute(.unique) because SwiftData isn't compatible.

I can't wait at launch either for the data to be pulled from the cloud as it takes time, the App doesn't know if there's data and the App would be empty if the device isnt connected.

For now the only thing that I've implemented is a sanity check at subsequent launches and when receiving a

@Query ( ... ) var data: [Data]

.onChange(of: data ...) 

If anyone has a better idea or more insight on SwiftData callbacks...

hi,

i've gone through this problem with an app in Core Data, and there are solutions available within CD by using different stores (one synched with the cloud and one not).

when SwiftData was introduced, i got the impression that creating different stores was supported ... and the API seems to be there ... but i've already seen several reports on the usual forums of multiple stores just not working yet. perhaps this will happen in SwiftData 2?

nevertheless, let me ask:

• will the data you pre-load be read-only? i.e., will the user have the ability to edit or delete any of it?
• will the user be able to create/edit/delete additional data?

let me know; i may be able to say more then.

DMG

Hey thanks for responding. Read only for the preloaded data. However the user can add user created entries with the same models but I do have flags to know if they were user created or pre-loaded.

I did some more testing like looking at how much time it takes to at least know if there's some data in the cloud but it's taking an eternity.

For example I tested a fresh install and no loading of any data but waiting for the cloud to sync and we are talking minutes. Minutes? The local storage is empty! My Json 555KB (the data that ended in the cloud)... I get there must be a bit of back and forth but still I don't get why they haven't optimized things in such a straight forward case...

I think I might have found a way. At launch I'm checking the number of records. Obviously if there's no connection etc ... but that's another issue.

```  private  func isFirstCloudSync() async throws -> Bool {

        let container = CKContainer.default()
        let cloudDB = container.privateCloudDatabase

        let pred = NSPredicate(value: true) //true -> return all records
        let query = CKQuery(recordType: "CD_Ingredient", predicate: pred)

        let (ingredientResults, _) = try await cloudDB.records(matching: query,
                                                            resultsLimit: 1)

        if ingredientResults.count == 0 {
            return true
        }
        return false

    }

hi @Michaf,

your situation seems quite similar to one i've worked on: you pre-load a number of read-only records (for us, about 450K of JSON), and the user can then perhaps create/edit/delete additional records "of the same type," as well as create records of another type that references those records. so i get what you're doing.

on the problem of finding out whether cloud data exists before pre-loading any JSON, i think this problem is not solvable. yes, your suggested solution works if you're connected and if the cloud is operating and if there is no significant latency and if ... well, that's a lot to hope for in general.

fact is, the cloud's going to do what it does when it wants to, and you have no control over that, so you really just do have to wait for it to do what it does.

so i think your solutions come down to either some form of de-duping (yuk), or else using separate SwiftData containers for what goes to the cloud and what does not.

(in my opinion) it's best to put only user-generated data in the cloud -- putting the same pre-loaded data into the cloud for every user seems wasteful -- and so using multiple containers is best.

unfortunately, SwiftData does not seem to do this right now. and, even if it did, you can't put some instances of a model type into one container and other instances of the same model type into a second container, since a model type is bound to a container. (but there is a way to work around this problem, as we have discovered).

as for any de-duping strategy, the problem will be deciding which of multiple records should be the real record to be kept in the cloud for all devices. what you might try is this:

• when pre-loading on any device from JSON, copy your own UUID from JSON, but also add a second UUID generated by the device to each model object (it need not be the same for each model object). this second UUID acts as a registration of the object being created on the device.
• if a device discovers multiple records with the same UUID as defined by your JSON, pick the one with the "lowest" value among the registration UUIDs (yeah, just sort by the uuidString values and pick the lowest) to be the record to keep in the cloud, and delete the others. over time, all devices will coalesce on the data being in the cloud.

as for how you discover that data has been duplicated, you're a little bit on your own. but for actually de-duping, you can group the records by the JSON-loaded UUID (see the dictionary method init(grouping:by:)), and apply the "pick the lowest" strategy to each value in the dictionary.

hope that's of use for you; feel free to contact me directly if you wish (i'm on GitHub)).

DMG

For the 2nd (device-generated) ID, perhaps a timestamp would be more useful than a UUID.

Thanks responding! I need to do some more testing. I'm also always suspicious of myself when I foresee a problem and overengineer a solution. For now the count check works when connnected. I wonder how many users will have it failed and end up with dups. I'm planning a Beta period so will get feedback!

Here are several strategies to address the data duplication issue in your app, combining insights from SwiftData's capabilities and best practices:

1. Conditional Inserts:

Before inserting data from the JSON, query SwiftData to check for existing entries with matching IDs. Only insert new items that don't already exist in the store, preventing duplicates.

2. Optimized Cloud Sync:

Prioritize Download: At launch, make fetching data from the cloud the primary action. Local Data as Fallback: If cloud sync fails or isn't available, use the JSON as a fallback. Implement Conflict Resolution: If data is modified on both devices offline, create conflict resolution mechanisms (e.g., choose the most recent version or prompt user for action).

3. Unique Identifier Generation:

If possible, modify the JSON generation process to create unique IDs (e.g., using timestamps or device identifiers). If not feasible, explore generating unique IDs within SwiftData itself before insertion.

4. Leveraging SwiftData Callbacks:

Utilize .onChange(of: data ...) to detect cloud data changes and update the local store accordingly, ensuring consistency. This can help maintain data integrity even when offline modifications occur. 5. Additional Considerations:

Manual ID Checks: If unique IDs in the JSON aren't possible, implement manual checks before insertion to identify potential duplicates. Enhanced Cloud Sync Strategies: Explore advanced sync patterns like real-time updates or conflict detection to further refine data management. ativadore.com

It seems like you might be referring to SwiftData, a data persistence library used in iOS development with Swift programming language. If you're experiencing issues related to duplicates when working with SwiftData, particularly in a database context, here are a few common reasons and solutions: www.quickprinting.ae

Primary Key Violations: Duplicates can occur if you're attempting to insert records with the same primary key. Ensure that the primary key is unique for each record. If there's a conflict, consider updating existing records instead of inserting new ones or handling the conflict gracefully.www.multiplemodapk.com/

Insertion Logic: Check the logic used for inserting data into the database. It's essential to avoid unintended duplication by verifying that you're not inadvertently inserting duplicate records.

Indexing and Constraints: Review the database schema and ensure that necessary indexing and constraints (unique constraints, foreign keys) are set appropriately to prevent duplicates.

Query and Fetch Operations: When fetching data, verify that your queries are correctly constructed to avoid returning duplicate records unintentionally. Review your SQL queries or SwiftData methods used for fetching data.

Data Integrity: Sometimes duplicates can arise due to issues with data integrity or synchronization. Make sure that your application's data management processes are robust and handle synchronization correctly if you're working with data from multiple sources.

Debugging and Logging: Implement logging and debugging mechanisms in your code to trace the flow of data and identify where duplicates might be introduced. This can help in pinpointing the source of the issue.

Database Cleanup: If you've identified duplicates, consider cleaning up the database by removing redundant or duplicate records. Ensure you have proper backups before performing any deletions.