Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind?

Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on.

But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?

  • MagicShel@programming.devEnglish
    32·
    2 years ago

    Pit Mutation testing is useful. It basically tests how effective your tests are and tells you missed conditions that aren’t being tested.

    For Java: https://pitest.org

    Edit: corrected to the more general name instead of a specific implementation.

    • mattburkedev@programming.devEnglish
      6·
      2 years ago

      The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases.

      Every enterprise I’ve consulted for that had code coverage requirements was full of elaborate mock-heavy tests with a single Assert.NotNull at the end. Basically just testing that you wrote the right mocks!

      • MagicShel@programming.devEnglish
        5·
        2 years ago

        That’s exactly the sort of shit tests mutation testing is designed to address. Believe me it sucks when sonar requires 90% pit test pass rate. Sometimes the tests can get extremely elaborate. Which should be a red flag for design (not necessarily bad code).

        Anyway I love what pit testing does. I hate being required to do it, but it’s a good thing.

    • vampatori@feddit.ukOPEnglish
      5·
      2 years ago

      This is really interesting, I’ve never heard of such an approach before; clearly I need to spend more time reading up on testing methodologies. Thank you!