new feature: provides a way to control memory allocation and deallocation

[tbEgg]

Feature Description

OpenDAL currently does not support allowing callers to take control of memory management, which is crucial for programs based on C and C++.

Problem and Solution

Enable users to define a custom memory allocator that can replace the default Rust memory allocation mechanisms.

Additional Context

No response

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.

[Xuanwo]

cc @PragmaTwice @xyjixyjixyji @silver-ymz, is it possible to pass an Allocator from the C side? I haven't seen such an attempt before; perhaps we are the pioneers in this area. It should be exciting.

[xyjixyjixyji]

I think it is possible, we pass 2/3 pointers (malloc, free, option<realloc>) to the rust side, and rust side set it as the global allocator. (api will look like opendal_set_malloc())

But user has to guarantee that if this is expected it should be called before any opendal internal function that might have dynamic memory allocation.

Another way would be we only expose several industry standard allocator directly as interfaces, such as opendal_set_use_jemalloc().