formatron.integrations.transformers Namespace Reference

This module integrates the transformers library by providing convenience utilities. More...

Classes
class	FormattersLogitsProcessor
	Logit processor that uses formatters to mask batch logits. More...

Functions
kbnf.Vocabulary	create_engine_vocabulary (PreTrainedTokenizerBase tokenizer, typing.Optional[list[typing.Callable]] vocab_processors=None)
	Create a vocabulary for the KBNF engine.
LogitsProcessor	create_formatter_logits_processor (PreTrainedTokenizerBase tokenizer, typing.Sequence[FormatterBuilder|None]|FormatterBuilder formatter_builders, typing.Sequence[EngineGenerationConfig] configs=None, typing.Optional[list[typing.Callable]] vocab_processors=None)
	Create a formatter logits processor.
LogitsProcessorList	create_formatter_logits_processor_list (PreTrainedTokenizerBase tokenizer, typing.Sequence[FormatterBuilder|None]|FormatterBuilder formatter_builders, typing.Sequence[EngineGenerationConfig] configs=None, typing.Optional[list[typing.Callable]] vocab_processors=None)
	Create a formatter logits processor list.

Detailed Description

This module integrates the transformers library by providing convenience utilities.

Function Documentation

create_engine_vocabulary()

kbnf.Vocabulary formatron.integrations.transformers.create_engine_vocabulary	(	PreTrainedTokenizerBase	tokenizer,
		typing.Optional[list[typing.Callable]]	vocab_processors = None )

Create a vocabulary for the KBNF engine.

Parameters

tokenizer	The tokenizer.
vocab_processors	List of callables with signature (token_to_char: typing.Dict[bytes, bytes])->None. Callables can be used to "unmangle" encoded characters to original characters. If None, processors will be auto-detected.

Definition at line 24 of file transformers.py.

create_formatter_logits_processor()

LogitsProcessor formatron.integrations.transformers.create_formatter_logits_processor	(	PreTrainedTokenizerBase	tokenizer,
		typing.Sequence[FormatterBuilder | None] | FormatterBuilder	formatter_builders,
		typing.Sequence[EngineGenerationConfig]	configs = None,
		typing.Optional[list[typing.Callable]]	vocab_processors = None )

Create a formatter logits processor.

Parameters

tokenizer	The tokenizer.
formatter_builders	The formatter builders.
configs	The engine generation configurations.
vocab_processors	List of callables with signature (token_to_char: typing.Dict[bytes, bytes])->None. Callables can be used to "unmangle" encoded characters to original characters. If None, processors will be auto-detected.

Definition at line 41 of file transformers.py.

create_formatter_logits_processor_list()

LogitsProcessorList formatron.integrations.transformers.create_formatter_logits_processor_list	(	PreTrainedTokenizerBase	tokenizer,
		typing.Sequence[FormatterBuilder | None] | FormatterBuilder	formatter_builders,
		typing.Sequence[EngineGenerationConfig]	configs = None,
		typing.Optional[list[typing.Callable]]	vocab_processors = None )

Create a formatter logits processor list.

Parameters

tokenizer	The tokenizer.
formatter_builders	The formatter builders.
configs	The engine generation configurations.
vocab_processors	List of callables with signature (token_to_char: typing.Dict[bytes, bytes])->None. Callables can be used to "unmangle" encoded characters to original characters. If None, processors will be auto-detected.

Definition at line 62 of file transformers.py.