Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local models? #2

Open
FinnT730 opened this issue Feb 14, 2024 · 16 comments
Open

Local models? #2

FinnT730 opened this issue Feb 14, 2024 · 16 comments

Comments

@FinnT730
Copy link

FinnT730 commented Feb 14, 2024

Will local models be supported one day as well?
(Unless they are, and I didn't find it in the readme XD)

@vyokky
Copy link
Contributor

vyokky commented Feb 15, 2024

Will local models be supported one day as well? (Unless they are, and I didn't find it in the readme XD)

That's on our todo list ;)

@FinnT730
Copy link
Author

👍 Thanks! Will wait for that.
Is it currently possible to use a program that emulates OpenAI's API? Something like Ollama?

@vyokky
Copy link
Contributor

vyokky commented Feb 15, 2024

👍 Thanks! Will wait for that. Is it currently possible to use a program that emulates OpenAI's API? Something like Ollama?

It only supports GPT-V for now. We plan to incorporate more models in the future.

@vyokky vyokky closed this as completed Feb 16, 2024
@vyokky vyokky reopened this Mar 9, 2024
@FinnT730

This comment was marked as off-topic.

@calamity10110
Copy link

I tried to edit config file:
OPENAI_API_BASE: "http://127.0.0.1:11434/" # The the OpenAI API endpoint,
OPENAI_API_KEY: "Null" # The API key
OPENAI_API_MODEL: "Llava"
result
Error making API request: Extra data: line 1 column 5 (char 4)
Error occurs when calling LLM.
Error making API request: Extra data: line 1 column 5 (char 4)
Error occurs when calling LLM.

for url http://127.0.0.1:11434/v1/chat/completions
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions
{'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}}
Error occurs when calling LLM.

@vyokky
Copy link
Contributor

vyokky commented Mar 24, 2024

Hi @calamity10110 , the current framework does not support non-openai model. We are working on it and will release a new feature for this soon.

@Justin-12138
Copy link

I am a non-subscriber of openai, can I still use UFO? I follow your instructions and my config file as following,I set the model to gpt3.5
version: 0.1
API_TYPE: "openai" # The API type, "openai" for the OpenAI API, "aoai" for the AOAI API.
OPENAI_API_BASE: "https://api.openai.com/v1/chat/completions" # The the OpenAI API endpoint, "https://api.openai.com/v1/chat/completions" for the OpenAI API.
OPENAI_API_KEY: "mykey" # The OpenAI API key
OPENAI_API_MODEL: "gpt-3.5-turbo-0301" # The only OpenAI model by now that accepts visual input
CONTROL_BACKEND: "uia" # The backend for control action

But I got error like this:
image

@FinnT730
Copy link
Author

FinnT730 commented Apr 6, 2024

Well, I am waiting for when local models can be used.
Right now, yes you need access to the API of OpenAPI, which is not free.

@vyokky
Copy link
Contributor

vyokky commented Apr 10, 2024

@FinnT730 @Justin-12138 You can now use models in Ollama for your local model deployment (in the pre-release branch). Please read https://github.com/microsoft/UFO/blob/pre-release/model_worker/README.md for details, and expect worse performance than GPT-V.

@vyokky
Copy link
Contributor

vyokky commented Apr 10, 2024

I tried to edit config file: OPENAI_API_BASE: "http://127.0.0.1:11434/" # The the OpenAI API endpoint, OPENAI_API_KEY: "Null" # The API key OPENAI_API_MODEL: "Llava" result Error making API request: Extra data: line 1 column 5 (char 4) Error occurs when calling LLM. Error making API request: Extra data: line 1 column 5 (char 4) Error occurs when calling LLM.

for url http://127.0.0.1:11434/v1/chat/completions Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error making API request: 400 Client Error: Bad Request for url: http://127.0.0.1:11434/v1/chat/completions {'error': {'message': 'json: cannot unmarshal array into Go struct field Message.messages.content of type string', 'type': 'invalid_request_error', 'param': None, 'code': None}} Error occurs when calling LLM.

@calamity10110 You can now use models in Ollama for Llava deployment (in the pre-release branch). Please read https://github.com/microsoft/UFO/blob/pre-release/model_worker/README.md for details, and expect worse performance than GPT-V.

@FinnT730
Copy link
Author

FinnT730 commented Apr 10, 2024 via email

@zsb87
Copy link

zsb87 commented May 23, 2024

Hello Team,

I tried using local model llava in pre-release branch, but unfortunately got this error. Did I miss anything here? Thanks

image

@vyokky
Copy link
Contributor

vyokky commented May 24, 2024

@Mac0q

@Mac0q
Copy link
Contributor

Mac0q commented May 24, 2024

@zsb87 It appears that your local model or API is refusing to respond. Usually this is because the model has limited functionality. Can you tell me your model version?

@zsb87
Copy link

zsb87 commented May 26, 2024

@Mac0q . This is my model version:
{"name":"llava:latest","model":"llava:latest","modified_at":"2024-05-20T13:50:45.2323374-07:00","size":4733363377,"digest":"8dd30f6b0cb19f555f2c7a7ebda861449ea2cc76bf1f44e262931f45fc81d081","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama","clip"],"parameter_size":"7B","quantization_level":"Q4_0"},"expires_at":"0001-01-01T00:00:00Z"}

@Mac0q
Copy link
Contributor

Mac0q commented May 26, 2024

@zsb87 I think llava:7b is still weak for this task, we will try it optimize the prompt to make it doable, but GPT-4V is for sure the best choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants