Skip to content

test: make test-process-argv-0 robust - #2541

Closed
Trott wants to merge 2 commits into
nodejs:masterfrom
Trott:test-argv0
Closed

Trott wants to merge 2 commits into
nodejs:masterfrom
Trott:test-argv0

Conversation

@Trott

@Trott Trott commented Aug 25, 2015

Copy link
Copy Markdown
Member

Remove use of STDERR to avoid test flakiness on CentOS.

Fixes: #2477

Additional benefit: This makes the test a fair bit faster.

Before:

real 0m0.909s
user 0m0.216s
sys 0m0.083s

After:

real 0m0.223s
user 0m0.193s
sys 0m0.035s

Remove use of STDERR to avoid test flakiness on CentOS.

Fixes: nodejs#2477
@Trott

Trott commented Aug 25, 2015

Copy link
Copy Markdown
Member Author

@ChALkeR ChALkeR added the test Issues and PRs related to Node.js core tests and test infrastructure. label Aug 25, 2015
@ChALkeR

ChALkeR commented Aug 25, 2015

Copy link
Copy Markdown
Member

But why was the test failing with stderr?

@Trott

Trott commented Aug 25, 2015

Copy link
Copy Markdown
Member Author

@ChALkeR I don't know. It's a good question. If there is a (CentOS-specific?) problem around stderr or something similar, that's a bug that deserves its own test and fix. Any ideas for zeroing in on that would be welcome.

That said, it's not a bug with process.argv so I'm comfortable removing it here.

Comment thread test/parallel/test-process-argv-0.js Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

common.mustCall

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how common.mustCall() can be used to replace the assertion here. Can you explain a bit more what you mean?

ghost Aug 25, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, common.mustCall makes sure that the exit handler is called for sure. Because the value is asserted only inside it. If the handler is never called, the test will still pass.

ghost Aug 25, 2015

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, because exit may not fire on the child process if there's an error. Good catch.

ghost Aug 25, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

common.mustCall uses process.exit internally. It is not safer than what is done here

ghost Aug 25, 2015

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually just going to just change child.on('exit', ...) to process.on('exit', ...) rather than use common.mustCall() for two reasons:

  • I think it's clearer what is going on that way.
  • This test does not load common.js to begin with because it doesn't need it. Loading the entire thing just for one use of common.mustCall() seems like overkill when process.on('exit', ...) will work just fine.

You (@targos) may very well be correct that child.on('exit', ...) and process.on('exit', ...) have the exact same behavior. But, at least according to the docs, the exit event on a ChildProcess might not fire. I can't find any similar warning in the docs on the exit event for process. So I'm inclined to go with process. If the exit event might not fire, that might be an issue to file against the docs. The text describing beforeExit seems to imply that exit can be depended on to fire for process.

ghost Aug 26, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, I misread the line as process.on('exit', ...).

@Trott

ghost commented Aug 25, 2015

Copy link
Copy Markdown
Member Author

Pushed some changes, so new CI: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/164/

@thefourtheye

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Contributor

I am really not sure how it actually works. There is nothing printed explicitly. But still data event gets invoked.

@joaocgreis

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member

As explained in the original commit, this test is just about argv[0]. My only concern is that the child might print the argv[0] and then die, and it really should not. However, the original test does nothing to check that, not even test the return value. Hence, the changes here do not break the test.

On the other hand, having the stderr of the child printed can be very useful for debug, it has just been now. The first commit is essentially a revert of 29463cb, it would be really nice to know why it was introduced but there is no PR-URL. And having child.stderr broken is a big issue - at the very least we must open an issue to find out what's wrong with the old test on CentOS.

@Trott did you have a chance to test this on CentOS? I may be able to test this in the next few days. If this really fixes the test, I wouldn't object landing and investigating the problem in another issue.

I'm +1 on the second commit, because if for some reason the child exit event callback does not run, the test may be false positive.

@Trott

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member Author

@joaocgreis I have not had a chance to run on CentOS other than the CI runs above. It would be great to see if 100 or 1000 consecutive runs of the old test produces a failure vs. results for the same with the new test. I suspect it won't be that simple and clear-cut though.

@Trott

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member Author

@joaocgreis I installed a CentOS 5 64-bit VM and managed to get the necessary versions of the right packages so that I can build node.

Running the test as it currently exists on the master branch over and over, it fails just like we see in the CI (with the same assertion failing in the same way) once in a while.

Running it in a loop until it blows up, I've had it fail as early as run 41 and make it as far as run 511.

Switching the version of the test in this PR, I can run it up to 10000 times and it does not blow up.

So I think we have a winner here.

For anyone with a CentOS 5 box wishing to replicate, here's the command I ran:

for i in {1..10000}; do echo $i; ./node test/parallel/test-process-argv-0.js || break; done;

@jbergstroem

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member

Just ran with/without these patches on our CI: centos5-32-1; without I managed to reproduce a few times below 100 runs while when applied (including coffee break) ran to 3257 without fail.

@thefourtheye

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Contributor

So the problem is with stderr?

@Trott

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member Author

I've opened an issue for the CentOS 5 issue that we're seeing in the current code.

Anyone feel comfortable giving this PR an official LGTM?

@targos

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member

LGTM

@Trott

ghost commented Aug 26, 2015

Copy link
Copy Markdown
Member Author

Landed in 972a57c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test Issues and PRs related to Node.js core tests and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants