huggingface pipeline 기능을 사용시에 truncate 하는 방법 (에러 : RuntimeError: The size of tensor a (1185) must match the size of tensor b (512) at non-singleton dimension 1))

•  발생 에러
• RuntimeError: The size of tensor a (1185) must match the size of tensor b (512) at non-singleton dimension 1
• input이 model input size보다 더커서 발생하는 문제
• tokenizer 선언히 truncate 설정해줬지만, pipeline 사용할 때는, pipeline 호출 할때, 다시 truncate를 설정해줘야 함

• 해결 방법

from transformers import pipeline
model_name = "test_trainer/checkpoint-423"
# load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained("klue/bert-base", padding=True, truncation=True, max_length=512)

classifier = pipeline("text-classification", model=model, tokenizer=tokenizer, device=0)
classifier(text, padding=True, truncation=True)