Skip to content

Issues: openai/tiktoken

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

36 Open 207 Closed
36 Open 207 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Tokenizer for o1
#363 opened Dec 23, 2024 by texasroh
The license file is not well referenced
#362 opened Dec 18, 2024 by tcosta-travelperk
Community Resource: AutoTikTokenizer - A Bridge Between TikToken and HuggingFace Tokenizers
#358 opened Nov 7, 2024 by bhavnicksm
2
Whisper
#354 opened Oct 21, 2024 by sine2pi
ValueError: not enough values to unpack (expected 2, got 1).
#353 opened Oct 17, 2024 by Itime-ren
3
Add aarch64 musllinux wheel for 0.8.0
#350 opened Oct 7, 2024 by rhuddleston
1
Tiktoken Permission denied error
#343 opened Sep 26, 2024 by NewGHUser4321
When I was inputting long text into a large model, that is, when the len of the text was 1024*1024, a StackOverflow error occurred.
#327 opened Jul 31, 2024 by YangQiangli
[FR] Add --offline
#317 opened Jul 6, 2024 by NightMachinery
3
TikToken Tokenizer from scratch ?
#303 opened May 25, 2024 by IsNoobgrammer
Tiktoken educational BPE trainer takes long time to train with vocab size 30k
#299 opened May 19, 2024 by sagorbrur
2
Use a custom exception ValueError subclass for the special tokens warning
#290 opened May 2, 2024 by simonw
Understanding the intended behaviour of _encode_bytes
#288 opened Apr 29, 2024 by ashleyholman
Optimize _byte_pair_merge function in BPE implementation
#284 opened Apr 12, 2024 by naveens01
how to convert qwen.tiktoken to tokenzier.model
#283 opened Apr 11, 2024 by cloudyuyuyu
SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url
#281 opened Apr 10, 2024 by sijiashen
8
How to find the token count of a prompt using meta/llama2-70b model
#274 opened Mar 29, 2024 by pradeepdev-1995
1
pinging tiktoken URL always fails, pinging api.openai.com always works
#269 opened Mar 14, 2024 by distributev
1
Unable to route GET requests through proxy
#259 opened Feb 15, 2024 by aevilevitch
2
Enhancement: Add convenience token-counting functions to this package
#250 opened Feb 7, 2024 by pamelafox
4
Panic (stack overflow) when encoding a certain string
#245 opened Jan 18, 2024 by Crazytieguy
7
Is there a way for tiktoken to interoperate better with offline AI software?
#232 opened Dec 27, 2023 by ParetoOptimalDev
3
F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer
#227 opened Dec 10, 2023 by Emasoft
2
tiktoken error on AWS lambda
#198 opened Sep 21, 2023 by vicky141998
5
Very slow for inputs like "a" * 100000
#195 opened Sep 18, 2023 by vvolhejn
10
ProTip! Mix and match filters to narrow down what you’re looking for.