Chat Options¶
Options are available for both Chat and AsyncChat.
Context¶
Context to start the chat with. The context must end on a system or an assistant message.
from llumpy.core import Chat, ConversationBuilder
from llumpy.providers import OllamaClient
chat = Chat(OllamaClient('llama3', '8b'), context=ConversationBuilder().system("Speak like a pirate"))
print(chat.message("Hello World!"))
print(chat.message("What was my last message"))
# Shiver me timbers! Yer last message be... "Hello World!" Arrr, don't ye worry about it not being a booty-ful greeting, matey! World was just fine with yer "hello"!
chat.clear_chat() # reset the chat back to the starting context
Context Window¶
A ContextWindow can be provided to limit the number of tokens an entire chat is allowed to use.
If a context window is exceeded, a ExceedContextWindowError is thrown.
from llumpy.core import Chat, ContextWindow
from llumpy.providers import AnthropicClient
claude = AnthropicClient('claude-sonnet-4-6')
chat = Chat(claude, context_window=ContextWindow(claude.tokenizer, max_tokens=1))
print(chat.message("Hello World!"))
# llumpy.core._chat.ExceedContextWindowError: Exceeded context window: requested 10 tokens, but only 1 are allowed
To check the remaining tokens, use the remaining_tokens property:
from llumpy.core import Chat, ContextWindow
from llumpy.providers import AnthropicClient
claude = AnthropicClient('claude-sonnet-4-6')
chat = Chat(claude, context_window=ContextWindow(claude.tokenizer, max_tokens=10))
print(chat.message("Hello World!"))
print(chat.remaining_tokens)
# Hello! 👋 Welcome! How are you doing today? Is there something I can help you with? 😊
# -26
Custom Tokenizers¶
Only Anthropic and OpenAI clients have the built in tokenizer methods since Ollama embeddings can vary.
Implement the Tokenizer class to use with the context window.
from llumpy.providers import OllamaClient
from llumpy.core import Chat, Tokenizer, History, ContextWindow
class Llama3Tokenizer(Tokenizer):
def __init__(self):
from transformers import AutoTokenizer # optional dependency
self._tok = AutoTokenizer.from_pretrained("llama3:8b")
def count_tokens(self, text: str | History) -> int:
if isinstance(text, str):
return len(self._tok.encode(text))
return len(self._tok.apply_chat_template(text.to_dicts(), tokenize=True, add_generation_prompt=True))
chat = Chat(
OllamaClient('llama3', '8b'),
context_window=ContextWindow(Llama3Tokenizer(), max_tokens=10)
)