• nogrub@lemmy.world
    link
    fedilink
    arrow-up
    39
    ·
    1 year ago

    it’s almost like computers are not that accurate when calculating floating point numbers

    • macrocephalic@lemmy.world
      link
      fedilink
      arrow-up
      4
      ·
      1 year ago

      It’s almost like some useless person created a variable with a distinct set unlikely to be higher than the hundreds as a floating point - when it obviously should have been an int.

      • WhiskyTangoFoxtrot@lemmy.world
        link
        fedilink
        arrow-up
        10
        ·
        1 year ago

        Nah, it makes sense to use a floating point number here, since unless the test is marked out of a factor of 100 then there will likely be a fractional value as the final percentage. The mistake was not rounding the final displayed value.