Hugging Face's logo

Transformers documentation

프로세서

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.19.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

프로세서

멀티모달 프로세서

모든 멀티모달 모델은 여러 모달리티(텍스트, 비전, 오디오)를 그룹화하는 데이터를 인코딩하거나 디코딩하는 객체가 필요한데, 이것은 프로세서라고 불리는 객체가 담당합니다. 프로세서는 토크나이저(텍스트 모달리티용), 이미지 프로세서(비전용), 특성 추출기(오디오용) 같이 두 개 이상의 처리 객체를 하나로 묶습니다.

이러한 프로세서는 저장 및 로딩 기능을 구현하는 다음 기본 클래스를 상속받습니다:

class transformers.ProcessorMixin

< >

( *args**kwargs )

This is a mixin used to provide saving/loading functionality for all processor classes.

apply_chat_template

< >

( conversation: list[dict[str, str]] | list[list[dict[str, str]]]chat_template: str | None = Nonetools: list[dict] | None = Nonedocuments: list[dict[str, str]] | None = Noneadd_generation_prompt: bool = Falsecontinue_final_message: bool | str = Falsereturn_assistant_tokens_mask: bool = Falsetokenize: bool = Falsereturn_tensors: str | transformers.utils.generic.TensorType | None = Nonereturn_dict: bool = Falseload_audio_from_video: bool = Falseprocessor_kwargs: dict | None = None**kwargs )

Parameters

  • conversation (Union[list[Dict, [str, str]], list[list[dict[str, str]]]]) — The conversation to format.
  • chat_template (Optional[str], optional) — The Jinja template to use for formatting the conversation. If not provided, the tokenizer’s chat template is used.

Similar to the apply_chat_template method on tokenizers, this method applies a Jinja template to input conversations to turn them into a single tokenizable string.

The input is expected to be in the following format, where each message content is a list consisting of text and optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form pixel_values when return_dict=True. If not provided, one will get only the formatted text, optionally tokenized text.

conversation = [ { “role”: “user”, “content”: [ {“type”: “image”, “url”: “https://www.ilankelman.org/stopsigns/australia.jpg”}, {“type”: “text”, “text”: “Please describe this image in detail.”}, ], }, ]

batch_decode

< >

( *args**kwargs )

This method forwards all its arguments to PreTrainedTokenizer’s batch_decode(). Please refer to the docstring of this method for more information.

check_argument_for_proper_class

< >

( argument_nameargument )

Checks the passed argument’s class against the expected transformers class. In case of an unexpected mismatch between expected and actual class, an error is raise. Otherwise, the proper retrieved class is returned.

create_mm_token_type_ids

< >

( input_ids: list ) → list[list[int]]

Parameters

  • input_ids (list[list[int]]) — Batch of token ID sequences. May be unpadded (variable length), so a plain Python list of lists is expected rather than a tensor or uniformly-shaped array.

Returns

list[list[int]]

A list of the same structure as input_ids, where each integer is the modality type ID for the corresponding token.

Build per-token modality type IDs for a batch of token_id sequences.

Each position is assigned an integer indicating which modality it belongs to: 0 for regular text, 1 for image tokens, 2 for video tokens, and 3 for audio tokens. Membership is determined by comparing against self.image_token_ids, self.video_token_ids, and self.audio_token_ids.

decode

< >

( *args**kwargs )

This method forwards all its arguments to PreTrainedTokenizer’s decode(). Please refer to the docstring of this method for more information.

from_args_and_dict

< >

( argsprocessor_dict: dict**kwargs ) → ~processing_utils.ProcessingMixin

Parameters

  • processor_dict (dict[str, Any]) — Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the ~processing_utils.ProcessingMixin.to_dict method.
  • kwargs (dict[str, Any]) — Additional parameters from which to initialize the processor object.

Returns

~processing_utils.ProcessingMixin

The processor object instantiated from those parameters.

Instantiates a type of ~processing_utils.ProcessingMixin from a Python dictionary of parameters.

from_pretrained

< >

( pretrained_model_name_or_path: str | os.PathLikecache_dir: str | os.PathLike | None = Noneforce_download: bool = Falselocal_files_only: bool = Falsetoken: str | bool | None = Nonerevision: str = 'main'**kwargs )

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — This can be either:

    • a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
    • a path to a directory containing a feature extractor file saved using the save_pretrained() method, e.g., ./my_model_directory/.
    • a path to a saved feature extractor JSON file, e.g., ./my_model_directory/preprocessor_config.json.
  • **kwargs — Additional keyword arguments passed along to both from_pretrained() and ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.

Instantiate a processor associated with a pretrained model.

This class method is simply calling the feature extractor from_pretrained(), image processor ImageProcessingMixin and the tokenizer ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the methods above for more information.

get_processor_dict

< >

( pretrained_model_name_or_path: str | os.PathLike**kwargs ) → tuple[Dict, Dict]

Parameters

  • pretrained_model_name_or_path (str or os.PathLike) — The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
  • subfolder (str, optional, defaults to "") — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here.

Returns

tuple[Dict, Dict]

The dictionary(ies) that will be used to instantiate the processor object.

From a pretrained_model_name_or_path, resolve to a dictionary of parameters, to be used for instantiating a processor of type ~processing_utils.ProcessingMixin using from_args_and_dict.

get_text_with_replacements

< >

( text: listimages_replacements: list = []videos_replacements: list = []audio_replacements: list = [] ) → tuple[list[str], list[dict[str, Any]]]

Parameters

  • text (list[str]) — Batch of raw text strings, each potentially containing multimodal placeholder tokens. Note that it will be modified in-place and returned.
  • images_replacements (list[str], optional, defaults to []) — Expanded replacement strings for each image, in the order they appear across the batch. Produced by self._process_images.
  • videos_replacements (list[str], optional, defaults to []) — Expanded replacement strings for each video. Produced by self._process_videos.
  • audio_replacements (list[str], optional, defaults to []) — Expanded replacement strings for each audio input. Produced by self._process_audio.

Returns

tuple[list[str], list[dict[str, Any]]]

A tuple of:

  • The modified text batch with all placeholder tokens expanded.
  • batch_replacement_offsets: one entry per batch item, each being a list of dicts with keys:
  • “type” (str): modality name — “image”, “video”, or “audio”
  • “span” (tuple[int, int]): original (start, end) char offsets of the placeholder token
  • “new_span” (tuple[int, int]): (start, end) offsets of placeholder in the expanded string
  • “text” (str): the original placeholder token string that was matched
  • “replacement” (str): the string it was replaced with

Replace multimodal placeholder tokens in a batch of text strings with their expanded representations, and return the modified texts alongside offset metadata.

This method is the core text-side preprocessing step for multimodal inputs. It scans each text in the batch for special tokens (image, video, audio) and replaces them in-order with the pre-computed replacement strings produced by self.replace_image_token / self.replace_video_token / self.replace_audio_token. Replacements are consumed from each modality’s list sequentially, so the i-th occurrence of e.g. self.image_token is replaced by images_replacements[i].

To add a new multimodal processor with placeholder tokens, you need to define a correct self.image_token which is the same token that is embedded in input text and also used as placeholder and repeated many times. Then you need to override self.replace_image_token to return the correct replacement string for a given image at index i. Same goes for all other supported modalities.

parse_response

< >

( response: str | list[int] | list[str] | list[list[int]] | np.ndarray | torch.Tensorschema: dict | None = Noneprefix: str | list[int] | list[str] | list[list[int]] | np.ndarray | torch.Tensor | None = Nonetools: list[dict | collections.abc.Callable] | None = None )

Parameters

  • response (str, token ids, 1D/2D tensor, or a list of these) — The output generated by the model, either decoded text or token ids, as a single sequence or a batch.
  • schema (dict, optional) — A response template. If not provided, the tokenizer’s response_template attribute is used.
  • prefix (str, token ids, 1D/2D tensor, or a list of these) — The prompt that came before generation. Many chat templates pre-write part of the message, so this is needed to parse correctly. For a batched response, pass either a single prefix (broadcast to every item) or one prefix per item. Only supported with new-style templates.
  • tools (list[Union[Dict, Callable]], optional) — Tools available to the model, in the same format as apply_chat_template accepts. Tool-call arguments are cast using the calling tool’s JSON schema.

Converts an output string created by generating text from a model into a parsed message dictionary. This method is intended for use with chat models, and will read the tokenizer’s response_template attribute to control parsing, unless a schema argument is passed directly.

Accepts either a single sequence (returning a single message dict) or a batch (returning a list of message dicts, one per item).

post_process_image_text_to_text

< >

( generated_outputsskip_special_tokens = True**kwargs ) → list[str]

Parameters

  • generated_outputs (torch.Tensor or np.ndarray) — The output of the model generate function. The output is expected to be a tensor of shape (batch_size, sequence_length) or (sequence_length,).
  • skip_special_tokens (bool, optional, defaults to True) — Whether or not to remove special tokens in the output. Argument passed to the tokenizer’s decode method.
  • **kwargs — Additional arguments to be passed to the tokenizer’s decode method.

Returns

list[str]

The decoded text.

Post-process the output of a vlm to decode the text.

post_process_multimodal_output

< >

( generated_outputsskip_special_tokens = Truegeneration_mode = None**kwargs ) → list[str]

Parameters

  • generated_outputs (torch.Tensor or np.ndarray) — The output of the model generate function. The output is expected to be a tensor of shape (batch_size, sequence_length) or (sequence_length,).
  • skip_special_tokens (bool, optional, defaults to True) — Whether or not to remove special tokens in the output. Argument passed to the tokenizer’s batch_decode method.
  • generation_mode (str, optional) — Generation mode indicated which modality to output and can be one of ["text", "image", "audio"].
  • **kwargs — Additional arguments to be passed to the tokenizer’s batch_decode method.

Returns

list[str]

The decoded text.

Post-process the output of a multimodal model to return the requested modality output. If the model cannot generated the requested modality, an error will be raised.

prepare_inputs_layout

< >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = Nonetext: str | list[str] | list[list[str]] | None = Nonevideos: typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = Noneaudio: typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None**kwargs: Unpack )

Normalize and prefetch inputs before processing. Wraps text in a list for multimodal processors, fetches remote images and audio if URLs are provided, and ensures audio is properly batched. Returns the normalized (images, text, videos, audio) tuple.

push_to_hub

< >

( repo_id: strcommit_message: str | None = Nonecommit_description: str | None = Noneprivate: bool | None = Nonetoken: bool | str | None = Nonerevision: str | None = Nonecreate_pr: bool = Falsemax_shard_size: int | str | None = '50GB'tags: list[str] | None = None )

Parameters

  • repo_id (str) — The name of the repository you want to push your processor to. It should contain your organization name when pushing to a given organization.
  • commit_message (str, optional) — Message to commit while pushing. Will default to "Upload processor".
  • commit_description (str, optional) — The description of the commit that will be created
  • private (bool, optional) — Whether to make the repo private. If None (default), the repo will be public unless the organization’s default is private. This value is ignored if the repo already exists.
  • token (bool or str, optional) — The token to use as HTTP bearer authorization for remote files. If True (default), will use the token generated when running hf auth login (stored in ~/.huggingface).
  • revision (str, optional) — Branch to push the uploaded files to.
  • create_pr (bool, optional, defaults to False) — Whether or not to create a PR with the uploaded files or directly commit.
  • max_shard_size (int or str, optional, defaults to "50GB") — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like "5MB").
  • tags (list[str], optional) — List of tags to push on the Hub.

Upload the processor files to the 🤗 Model Hub.

Examples:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("google-bert/bert-base-cased")

# Push the processor to your namespace with the name "my-finetuned-bert".
processor.push_to_hub("my-finetuned-bert")

# Push the processor to an organization with the name "my-finetuned-bert".
processor.push_to_hub("huggingface/my-finetuned-bert")

register_for_auto_class

< >

( auto_class = 'AutoProcessor' )

Parameters

  • auto_class (str or type, optional, defaults to "AutoProcessor") — The auto class to register this new feature extractor with.

Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with AutoProcessor.

save_pretrained

< >

( save_directorypush_to_hub: bool = False**kwargs )

Parameters

  • save_directory (str or os.PathLike) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist).
  • push_to_hub (bool, optional, defaults to False) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with repo_id (will default to the name of save_directory in your namespace).
  • kwargs (dict[str, Any], optional) — Additional key word arguments passed along to the push_to_hub() method.

Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the from_pretrained() method.

This class method is simply calling save_pretrained() and save_pretrained(). Please refer to the docstrings of the methods above for more information.

to_dict

< >

( ) → dict[str, Any]

Returns

dict[str, Any]

Dictionary of all the attributes that make up this processor instance.

Serializes this instance to a Python dictionary.

to_json_file

< >

( json_file_path: str | os.PathLike )

Parameters

  • json_file_path (str or os.PathLike) — Path to the JSON file in which this processor instance’s parameters will be saved.

Save this instance to a JSON file.

to_json_string

< >

( ) → str

Returns

str

String containing all the attributes that make up this feature_extractor instance in JSON format.

Serializes this instance to a JSON string.

validate_inputs

< >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor'], NoneType] = Nonetext: str | list[str] | list[list[str]] | None = Nonevideos: typing.Union[list['PIL.Image.Image'], numpy.ndarray, ForwardRef('torch.Tensor'), list[numpy.ndarray], list['torch.Tensor'], list[list['PIL.Image.Image']], list[list[numpy.ndarray]], list[list['torch.Tensor']], transformers.video_utils.URL, list[transformers.video_utils.URL], list[list[transformers.video_utils.URL]], transformers.video_utils.Path, list[transformers.video_utils.Path], list[list[transformers.video_utils.Path]], NoneType] = Noneaudio: typing.Union[numpy.ndarray, ForwardRef('torch.Tensor'), collections.abc.Sequence[numpy.ndarray], collections.abc.Sequence['torch.Tensor'], NoneType] = None**kwargs: Unpack )

Validate that at least one input is provided and that no deprecated keyword arguments are used. Raises ValueError otherwise.

Override when the processor needs additional validation on the input args.

Update on GitHub

←파이프라인 양자화→