Skip to main content

different data connectors for different objects and users [Resolved]

I have an application which needs to store different objects in different data stores based on the user's authorization credentials. I.E. the user may need to save object 1 to their BI postgres database, object 2 goes to their S3 bucket, and object 3 goes somewhere else.

Inversely, they may want to read data from different datastores based on these credentials as well. I.E records from the union of a few databases contingent on some predicate "get all rows in db1 and db2 such that the rows in db2 aren't in db3"

I'm wondering if anyone knows a good approach or design for this type of use case

Question Credit: Anisotropic
Question Reference
Asked April 16, 2019
Posted Under: Programming
1 Answers

I don't know if there is such a system (I hardly doubt it, where your application is so specific to your needs pg + S3 + "Somewhere Else").

First you should create a middle layer to enforce authorization/authentication:
It may depend on the query requested and the nature of the workload you want to provide.
For example; do the users interact with your service in a request/response matter (and providing the credentials in the request) or the users interact in a session (providing all of the credentials and transact with the service over a period of time) ?

Secondly, you should strive to expose a uniform API to the user for interacting with your system through it. The user shouldn't know your implementation details and you should encapsulate it from him.
You may support only a limited functionality due to this abstraction and you'll have to translate the queries accordingly. (i.e. running such predicate as you mention where one data store is pg and the other data store is S3 may be non-trivial in some settings)

Another option is to consolidate your operation to a single data-store system (for instance limit yourself to pg) use its' expressive API / authorization facilities etc.
And only use other data stores (w/o combining them in such inter-queries) for limited purposes (for example, cold storage of logs).

(Best of luck!)

credit: nadir
Answered April 16, 2019
Your Answer