My first Swift app is a photo library manager, and after rebuilding its Core Data guts half a dozen times, I am throwing up my hands and asking for help. For each photo, there are a few "layers" of work I need to accomplish before I can display it:

1. Create an Asset (NSManagedObject subclass) in Core Data for each photo in the library.
2. Do some work on each instance of Asset.
3. Use that work to create instances of Scan, another NSManagedObject class. These have to-many relationships to Assets.
4. Look over the Scans and use them to create AssetGroups (another NSManagedObject) in Core Data. Assets and AssetGroups have many-to-many relationships.

For each photo, each layer must complete before the next one starts. I can do multiple photos in parallel, but I also want to chunk up the work so it loads into the UI coherently.

I'm really having trouble making this work gracefully; I've built and rebuilt it a bunch of different ways. My current approach uses singleton subclasses of this Service, but as soon as I call save() on the first one, the work stops.

Service.swift

class Service: NSObject, ObservableObject, NSFetchedResultsControllerDelegate {

    var name: String
    var predicate: NSPredicate
    var minStatus: AssetStatus
    var maxStatus: AssetStatus
    internal let queue: DispatchQueue
    internal let mainMOC = PersistenceController.shared.container.viewContext
    internal let privateMOC = PersistenceController.shared.container.newBackgroundContext()
    internal lazy var frc: NSFetchedResultsController<Asset> = {
        let req = Asset.fetchRequest()
        req.predicate = self.predicate
        req.sortDescriptors = [NSSortDescriptor(key: #keyPath(Asset.creationDate), ascending: false)]
        let frcc = NSFetchedResultsController(fetchRequest: req,
                                              managedObjectContext: self.mainMOC,
                                              sectionNameKeyPath: "creationDateKey",
                                              cacheName: nil)
        frcc.delegate = self
        return frcc
    }()
    @Published var isUpdating = false
    @Published var frcCount = 0

    init(name: String, predicate: NSPredicate? = NSPredicate(value: true), minStatus: AssetStatus, maxStatus: AssetStatus) {
        self.name = name
        self.predicate = predicate!
        self.minStatus = minStatus
        self.maxStatus = maxStatus
        self.queue = DispatchQueue(label: "com.Ladybird.Photos.\(name)", attributes: .concurrent)
        super.init()
        self.fetch()
        self.checkDays()
    }

    private func fetch() {
        do {
            try self.frc.performFetch()
            print("\(name): FRC fetch count: \(frc.fetchedObjects!.count)")
        } catch {
            print("\(name): Unable to perform fetch request")
            print("\(error), \(error.localizedDescription)")
        }
    }

    func savePrivate() {
        self.privateMOC.perform {
            do {
                try self.privateMOC.save()

                }
             catch {
                print("\(self.name) could not synchonize data. \(error), \(error.localizedDescription)")
            }
        }
    }

    func save() {

        do {
            try self.privateMOC.save()

            self.mainMOC.performAndWait {
                do {
                    try self.mainMOC.save()
                } catch {
                    print("\(self.name) could not synchonize data. \(error), \(error.localizedDescription)")
                }
            }
        }

             catch {
                print("\(self.name) could not synchonize data. \(error), \(error.localizedDescription)")
            }
    }

    func checkDays() {
        // Iterate over days in the photo library
        if self.isUpdating { return }
        self.isUpdating = true
//        self.updateCount = self.frcCount
        var daysChecked = 0
        var day = Date()
        while day >= PhotoKitService.shared.oldestPhAssetDate() {
            print("\(name) checkDay \(DateFormatters.shared.key.string(from: day))")

            checkDay(day)
            var dc = DateComponents()
            dc.day = -1
            daysChecked += 1
            day = Calendar.current.date(byAdding: dc, to: day)!
            if daysChecked % 100 == 0 {
                DispatchQueue.main.async {
                    self.save()
                }
            }
        }
        self.save()
        self.isUpdating = false
    }

    func checkDay(_ date: Date) {
        let dateKey = DateFormatters.shared.key.string(from: date)
        let req = Asset.fetchRequest()
        req.predicate = NSPredicate(format: "creationDateKey == %@", dateKey)
        guard let allAssetsForDateKey = try? self.mainMOC.fetch(req) else { return  }

        if allAssetsForDateKey.count == PhotoKitService.shared.phAssetsCount(dateKey: dateKey) {
            if allAssetsForDateKey.allSatisfy({$0.assetStatusValue >= minStatus.rawValue && $0.assetStatusValue <= maxStatus.rawValue}) {
                let frcAssetsForDateKey = self.frc.fetchedObjects!.filter({$0.creationDateKey! == dateKey})
                if !frcAssetsForDateKey.isEmpty {
                    print("\(name): Day \(dateKey) ready for proccessing.")
                    for a in frcAssetsForDateKey {
                        self.handleAsset(a)
                    }
                }
            }
        }
        self.save()
    }

    // implemented by subclasses
    func handleAsset(_ asset: Asset) -> Void { }

    func controllerDidChangeContent(_ controller: NSFetchedResultsController<NSFetchRequestResult>) {
        self.frcCount = self.frc.fetchedObjects?.count ?? 0
        self.checkDays()
    }

}

I have a subclass of this for each of the four steps above. I want the data to flow between them nicely, and in previous implementations it did, but I couldn't chunk it the way I wanted, and it crashed randomly. This feels more controllable, but it doesn't work: calling save() stops the iteration happening in checkDays(). I can solve that by wrapping save() in an async call like DispatchQueue.main.async(), but it has bad side effects — checkDays() getting called while it's already executing. I've also tried calling save() after each Asset is finished, which makes the data move between layers nicely, but is slow as hell.

So rather than stabbing in the dark, I thought I'd ask whether my strategy of "service layers" feels sensible to others who others who have dealt with this kind of problem. It'd also be helpful to hear if my implementation via this Service superclass makes sense.

What would be most helpful is to hear from those with experience how they would approach implementing a solution to this problem: consecutive steps, applied concurrently to multiple Core Data entities, all in the background. There are so many ways to solve pieces of this in Swift — async/await, Tasks & Actors, DispatchQueues, ManagedObjectContext.perform(), container.performBackgroundTask(), Operation… I've tried each of them to mixed success, and what I feel like I need here is a trail map to get out of the forest.

Thanks y'all

I had a smiliar problem a while ago. I can't test your code because I don't have time. From the provided code I can see you only have one queue (DispatchQueue). Even when it has the attribute concurrent it's only one queue. Then you have your service as a Singleton, so it's only one queue. At least that's how I understand it.

I created for me not a DispatchQueue but an OperationQueue where I put all my operations (in your case 4?) in and let them run on this OperationQueue in the background. I don't know if this helps you. I'm sure there is a way to solve this problem with async await as well but I'm not yet familiar with it.