Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Implement async sync_files method in data loaders in python SDK #1007

Open
Default2882 opened this issue Nov 6, 2024 · 1 comment
Open

Comments

@Default2882
Copy link
Contributor

Problem?

In python-sdk/data_loaders, we have url_loader.py and local_directory_loader.py. We need to add another method sync_files to the existing data loaders which will return a Generator of the file. The requirement is to write it in an async way so that indexify can poll it internally.

User DX -

The function should look like -

def sync_files(s3_bucket: str) -> Generator[File]:
  ...
  yield File(...)

Testing -

TBD

@Default2882
Copy link
Contributor Author

@diptanu Please take a look at this issue's description, and also assign it to me. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant