How to provide an API client with 1,000,000 database results?
- by Chris Dutrow
What is a good way to provide an API client with 1,000,000 database results?
We are cureently using PostgreSQL. A few suggested methods:
Paging using Cursors
Paging using random numbers ( Add "GREATER THAN ORDER BY " to each query )
Save information to a file and let the client download it
Iterate through results, then POST the data to the client server
Return only keys to the client, then let the client request the objects from Cloud files like Amazon S3 (still may require paging just to get the file names ).
What haven't I thought of that is stupidly simple and way better than any of these options?