NASA: on testing versus multiple implementations

Here’s an short piece from NASA on three approaches to designing high reliability systems:

  • testing
  • multiple independent implementations
  • formal methods

It concludes that formal methods are the only way to achieve very high reliability because 1) rigorous testing takes too long, and 2) multiple implementations tend not to work, and can never be show to work because testing to compare would take too long.

It says multiple implementations will give the illusion of extra reliability, but that data suggests otherwise. So for us testing is good :-).

https://shemesh.larc.nasa.gov/fm/fm-why.html

7 Likes

Thank you for sharing this piece.

I’m so curious to learn more.

I found this paper on formal methods: https://users.ece.cmu.edu/~koopman/des_s99/formal_methods/

Does anyone have experience with applying formal methods? I’d like to see how it practically works.

The paper mentions that engineers need to be versed in type theory to properly apply formal methods. Sounds like Rust may be developed in this manner. I’m going to ask around…

1 Like