fix: multiword broken

[stephantul]

On current release, multiword distillation is gone. Any multi-word tokens just get removed from the vocabulary, i.e., nothing breaks or is broken, users are just not able to add multi-word tokens.

This PR fixes that. The issue was that we add multiword tokens as added tokens, but call prune_added_tokens after adding the tokens. We need to call prune_added_tokens to get rid of broken EOS/BOS tokens in some tokenizers. This slipped past unit testing coverage because we did not have an end to end test with multiword units, and the test that tested multiword units only covered the actual creation of the vocabulary.

I backported the test to main, and saw it failed there. So the added test is a test of the issue we're solving.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
model2vec/distill/distillation.py	90.14% <ø> (-0.28%)	⬇️
model2vec/tokenizer/tokenizer.py	100.00% <100.00%> (ø)
model2vec/version.py	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[stephantul]

Multiword tokens? More like multiword brokens

🤣       🤣🤣🤣🤣  🤣
🤣       🤣    🤣  🤣
🤣       🤣    🤣  🤣
🤣       🤣    🤣  🤣
🤣🤣🤣🤣  🤣🤣🤣🤣  🤣🤣🤣🤣