formatron.integrations.exllamav2 Namespace Reference

This module integrates the ExLlamaV2 library by providing convenience utilities. More...

Classes
class	FormatterFilter
	ExLlamaV2Filter that uses a formatter to mask logits. More...

Functions
kbnf.Vocabulary	create_engine_vocabulary (ExLlamaV2Tokenizer tokenizer, typing.Optional[list[typing.Callable]] vocab_processors=None)
	Create a vocabulary for the KBNF engine.
ExLlamaV2Filter	create_formatter_filter (ExLlamaV2 model, ExLlamaV2Tokenizer tokenizer, FormatterBuilder formatter_builder, EngineGenerationConfig engine_config=None, typing.Optional[list[typing.Callable]] vocab_processors=None)
	Create a formatter filter for the ExLlamaV2 engine.

Detailed Description

This module integrates the ExLlamaV2 library by providing convenience utilities.

Function Documentation

create_engine_vocabulary()

kbnf.Vocabulary formatron.integrations.exllamav2.create_engine_vocabulary	(	ExLlamaV2Tokenizer	tokenizer,
		typing.Optional[list[typing.Callable]]	vocab_processors = None )

Create a vocabulary for the KBNF engine.

Parameters

tokenizer	The tokenizer.
vocab_processors	List of callables with signature (token_to_char: typing.Dict[bytes, bytes])->None. Callables can be used to "unmangle" encoded characters to original characters. If None, processors will be auto-detected.

Definition at line 24 of file exllamav2.py.

create_formatter_filter()

ExLlamaV2Filter formatron.integrations.exllamav2.create_formatter_filter	(	ExLlamaV2	model,
		ExLlamaV2Tokenizer	tokenizer,
		FormatterBuilder	formatter_builder,
		EngineGenerationConfig	engine_config = None,
		typing.Optional[list[typing.Callable]]	vocab_processors = None )

Create a formatter filter for the ExLlamaV2 engine.

Parameters

model	The ExLlamaV2 model.
tokenizer	The ExLlamaV2 tokenizer.
formatter_builder	The formatter builder.
engine_config	The engine generation configuration.
vocab_processors	List of callables with signature (token_to_char: typing.Dict[bytes, bytes])->None. Callables can be used to "unmangle" encoded characters to original characters. If None, processors will be auto-detected.

Definition at line 46 of file exllamav2.py.