It’s just data

Need for More System Tests

Rails 2.3.6, 2.3.7, 2.3.8, all in less than 48 hours.  My tests played a role in getting rails back on course.  Overall, I continue to believe that it is much more important for a project to be responsive than correct.  Ideally, however, these problems would have been noted before shipping.  At least two corrections make sense in response:

A slight correction (I’m the author of Haml): the bugs in Rails 2.3.6 were not limited to its interactions with Haml and other alternate templating engines, although there were bugs of that sort as well. The reason 2.3.7 was released so soon was that my investigation into the bugs affecting Haml turned up some serious security issues which affected at least all users of Rails' XSS protection.

In addition, I wouldn’t say that rails_xss caused the issues. The way the XSS-protection backport works is that most of the XSS-protection code is in Rails proper (which incidentally worked fine in Rails 2.3.5), and rails_xss just activates it. The bugs were caused by a modification of the XSS protection in Rails, and indeed affected Haml whether or not rails_xss was installed. It is true that the security bug that necessitated 2.3.7 only affected rails_xss users, though.

Posted by Nathan Weizenbaum at

Nathan: thanks for the clarification.  From my perspective, none of this obviates the need for more system testing prior to releases of Rails.  I personally don’t believe that having the Rails team testing all possible permutations scales.

The discussion I would like to motivate: what can be done to identify potential breakages to both the core team and to library maintainers?  Ideally this would be done closer to the time the change is made, but in any case before a release is made?

My experience is that if I communicate a problem to the core team within a few days of when the change was introduced, problems will be addressed within a few days.  Often this feedback loop is much, much tighter than that: often I can report a problem within hours and the fix is made within hours.

Posted by Sam Ruby at

Third Time's the Charm?

Tribulations updating Instiki for Rails 2.3.8....

Excerpt from Musings at

I agree that more system tests would be good, but in this case finding at least the issues with 2.3.6 would have been more straightforward: if there were any tests for rails_xss, they would have revealed these issues. There were several layers of redundancy already in place that should have caught these issues. Wider testing would provide another layer to catch more issues faster, which is great, but I think it’s important to recognize that this change would have been caught by the standard methods of detecting errors had they been applied.

Posted by Nathan Weizenbaum at

Having tests for rails_xss would have stopped 2.3.6 from being released.

What tests would have prevented 2.3.7 from being released?

Posted by Sam Ruby at

Add your comment