• barsoap@lemm.ee
    link
    fedilink
    English
    arrow-up
    4
    ·
    8 months ago

    And CPUs still do it to this day. Nasty, nasty maths involved in figuring out an optimal combination between lookup table size and refinement calculations because that output can’t be approximate, it has to work how IEEE floats are supposed to work. Pure numerology.