• Pieisawesome@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 day ago

    It’s because LLMs don’t work with letters. They work with tokens that are converted to vectors.

    They literally don’t see the word “strawberry” in order to count the letters.

    Splitting the letter probably separates them into individual tokens