Skip to content

Retry failed benchmark result pushes - #488

Merged
savannahostrowski merged 4 commits into
python:mainfrom
savannahostrowski:fix/benchmark-push-retries
Nov 17, 2026
Merged

savannahostrowski merged 4 commits into
python:mainfrom
savannahostrowski:fix/benchmark-push-retries

Conversation

@savannahostrowski

@savannahostrowski savannahostrowski commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Another cherry-pick from my fork, with a few adjustments for upstream :)

A few months back, I encountered failed result pushes when running benchmarks across multiple machines, causing jobs to fail after the benchmarks had completed.

This PR adds up to five push attempts using EndBug/add-and-commit’s built-in push_attempts, with --rebase --autostash pulls between failed attempts. It covers benchmark results, pystats, derived results, and failure reports.

@maurycy maurycy left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two notes, maybe as a follow-up?

  • What do you think about just adding a new command to the bench runner? Say: python -m bench_runner push_results. This seems to be in line with
    ### Moving more code to Python
  • This seems to touch only push, not touching pull.

@maurycy maurycy left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that the latest EndBug/add-and-commit has push_attempts:

https://github.com/EndBug/add-and-commit/blob/cc9c08ba6c8df3b93a8f2db63e89b98368ae2ae8/README.md?plain=1#L162-L167

The problem is that it's a bit forceful:

https://github.com/EndBug/add-and-commit/blob/cc9c08ba6c8df3b93a8f2db63e89b98368ae2ae8/src/main.ts#L212-L242

We've got a backoff, and they do not sleep etc. Thinking about submitting a PR there

On the other hand:

with:
  pull: --rebase --autostash origin main
  push_attempts: 5

looks so nice.

@savannahostrowski

Copy link
Copy Markdown
Member Author

ooo, I do like using their built-in retry better. It's probably fine to go that route. Would be nice if they had sleep but it might be enough just with raw retries?

@maurycy

maurycy commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

@savannahostrowski My hunch it's fine til we start processing a significant number of cpython PRs (then we definitely run into more concurrency-related issues here; the whole thing is hard to follow), and it's much simpler!

@savannahostrowski

Copy link
Copy Markdown
Member Author

Thanks! I’ve updated to use the action’s built-in retries.

I included --autostash and looked into the existing -X ours. #323 already tracks problems with that pull’s conflict handling; an attempted fix was reverted because it broke on Windows.

@savannahostrowski
savannahostrowski merged commit 1e0cbf3 into python:main Nov 17, 2026
9 checks passed
@maurycy

maurycy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

@savannahostrowski This is great. Thank you for checking my review so carefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants