formatron.integrations._utils Namespace Reference

Functions
	_multiple_replace (replacements, regex, text)
	_autodetect_processors (typing.Dict[str, int] vocab)
typing.Dict[bytes, int]	get_original_characters (typing.Dict[str, int] vocab)
	huggingface_bytelevel_decoder ()
	I hate legacy code.

Variables
	Processors = set[typing.Literal["sentencepiece", "<0xHH>", "dot_G"]]

Function Documentation

_autodetect_processors()

formatron.integrations._utils._autodetect_processors ( typing.Dict[str, int] vocab )

protected

Definition at line 14 of file _utils.py.

_multiple_replace()

formatron.integrations._utils._multiple_replace ( replacements, regex, text )

protected

Definition at line 6 of file _utils.py.

get_original_characters()

typing.Dict[bytes, int] formatron.integrations._utils.get_original_characters

(

typing.Dict[str, int]

vocab

)

Definition at line 28 of file _utils.py.

huggingface_bytelevel_decoder()

formatron.integrations._utils.huggingface_bytelevel_decoder

(

)

I hate legacy code.

Definition at line 55 of file _utils.py.

Variable Documentation

Processors

formatron.integrations._utils.Processors = set[typing.Literal["sentencepiece", "<0xHH>", "dot_G"]]

Definition at line 11 of file _utils.py.