Visualizer
译者：片刻小哥哥
项目地址：https://huggingface.apachecn.org/docs/tokenizers/api/visualizer
Annotation
class
tokenizers.tools.
Annotation
[<
source
](https://github.com/huggingface/tokenizers/blob/v0.13.4.rc2/src/tokenizers/tools/visualizer.py#L16)
(
start
: int
end
: int
label
: str
)
EncodingVisualizer
class
tokenizers.tools.
EncodingVisualizer
[<
source
](https://github.com/huggingface/tokenizers/blob/v0.13.4.rc2/src/tokenizers/tools/visualizer.py#L67)
(
tokenizer
: Tokenizer
default_to_notebook
: bool = True
annotation_converter
: typing.Union[typing.Callable[[typing.Any], tokenizers.tools.visualizer.Annotation], NoneType] = None
)
Parameters
- tokenizer ( Tokenizer ) — A tokenizer instance
- default_to_notebook
(
bool) — Whether to render html output in a notebook by default - annotation_converter
(
Callable, optional ) — An optional (lambda) function that takes an annotation in any format and returns an Annotation object
Build an EncodingVisualizer
__call__
[<
source
](https://github.com/huggingface/tokenizers/blob/v0.13.4.rc2/src/tokenizers/tools/visualizer.py#L108)
(
text
: str
annotations
: typing.List[tokenizers.tools.visualizer.Annotation] = []
default_to_notebook
: typing.Optional[bool] = None
)
Parameters
- text
(
str) — The text to tokenize - annotations
(
List[Annotation], optional ) — An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function - default_to_notebook
(
bool, optional , defaults to False ) — If True, will render the html in a notebook. Otherwise returns an html string.
Build a visualization of the given text
