Opinions requested- Developing a scientific macOS App in SwiftUI that incorporates 250 x 300kb data sets. Which is the better option 1) One large JSON file with 250 items, each 300kb or 2) 250 JSON files, each with 300kb of data?

The bulk of the data are arrays of Int64 with 100,000 elements.

@steve has two approaches to reading data:

Which is the better option?

Ha! Your question reminds me of the meme: Would you rather fight one horse-sized duck, or 100 duck-sized horses?

Instruments

I've wanted to know more about xcode's Instruments application. You could profile both approaches and graphically see which approach best suits your operational needs.

I have no proof, but I would think the overhead of locating 250 JSON files, checking read/write access controls to each file, opening 250 files, and reading the contents, then closing 250 files would add more to your app than opening one larger file. Again, I have no proof, but this seems like a question that instruments could quickly answer!

Please return and let us know your results.

Thank you @Obelix. I like the analogy. I will probably implement the first option with one large JSON file and if that works call it good.

Honestly, I don't think JSON is the right solution for this. Codable is notoriously slow and loading that number of records would probably be excruciating. If you are definitely going to stick with Codable, though, I would think a number of smaller files would be better than one big one.

Benchmarked with 10 data records (ecah about 450 KB) in a single JSON file - program loaded and ran in 3 seconds on a Mac Studio Max. Extrapolating to 250 records, loading will take 68 seconds. That is probably too long? As Roosterboy stated, 250 smaller JSON files might be better. Since I do not need all 250 records in memory at once, could load each JSON file as needed by user?