Some run-tests improvements by stinos · Pull Request #18803 · micropython/micropython

stinos · 2026-02-10T15:34:08Z

Summary

Fix confusing issues like

>cd micropython/tests
>./run-tests.py -d extmod
...
180 tests passed
22 tests skipped

>./run-tests.py -d ./extmod
...
180 tests passed
21 tests skipped
1 tests failed: ./extmod/ssl_poll.py

by comparing full paths for the skip_tests logic.

Fix

>./run-tests.py -d .
...
FAIL ./run-internalbench.py
<hangs, probably because of recursively calling into run-tests.py again>

by running the normal test suite since the directory passed is our own tests/ directory.

Make test display and output consistent, such that

>./run-tests.py
...
pass extmod/btree1.py
...

>./run-tests.py -d ./extmod
...
pass ./extmod/btree1.py
...

>./run-tests.py -d /full/path/to/extmod
...
pass /full/path/to/extmod/btree1.py
...

instead will always use the first form extmod/btree1.py, both in display and results.json, and in case of failure will also always output the same .out/.exp files (so not e.g. extmod_btree1.py.exp for the first case but _full_path_to_extmod_btree1.py.exp for the second case).

Add a --dry-run argument to show which test files would be ran (only implemented for run-tests, not run-multitests etc, would that be wanted?):

> ./run-tests.py --dry-run -d ./extmod
...
found ./extmod/re_span.py
found ./extmod/json_loads_int_64.py
skip  ./extmod/ssl_poll.py
found ./extmod/select_poll_basic.py
...

Testing

Commands shown above and in the commit message were all tested successfully on unix and windows ports.

Trade-offs and Alternatives

Trade-off: code is somewhat more complex, realpath might hit filesystem and be slower.

Alternative implementation: the overall proper fix would probably be to put any path which enters or is used in run-tests.py to be wrapped in pathlib.Path(<somepath>).resolve() (maybe even add an expanduser there) so any comparison just works and the various realpath/abspath calls and having to differentiate test_file/test_file_abspath/... can be gone.

Alternative usage: learn to use run-tests.py 'properly' as to workaround these issues (I for one can apparently not do that because I've ran into this quite way too much :) )

codecov · 2026-02-10T15:41:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.42%. Comparing base (9721955) to head (6fc41e1).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #18803   +/-   ##
=======================================
  Coverage   98.42%   98.42%           
=======================================
  Files         174      174           
  Lines       22333    22333           
=======================================
  Hits        21982    21982           
  Misses        351      351

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Josverl · 2026-02-10T18:47:37Z

learn to use run-tests.py 'properly' as to workaround these issues

Can you elaborate on this ?

Josverl · 2026-02-10T18:47:38Z

learn to use run-tests.py 'properly' as to workaround these issues

Can you elaborate on this ?

andrewleech · 2026-02-10T20:38:55Z

Can you elaborate on this ?

I presume he just means to not "accidentally" add a leading ./ to the domain arg!

This change looks straightforward enough to me, a quick dry run to see what tests are going to be run can be useful before starting a bigger test run.

Is skip tests logic the only thing that could get confused by the different path syntax?

Should the -d arg be passed through realpath immediately to verify it and make it consistently formatted for all things that use it?

Signed-off-by: stijn <stijn@ignitron.net>

This is convenient when trying to figure out the correct values for --include/--exclude/--test-dirs/... arguments. Signed-off-by: stijn <stijn@ignitron.net>

Test file paths which get passed to the run_tests function can be absolute or relative and with or without leading slash in the latter case, depending on the arguments to run-tests.py, but the skip_tests list with tests to skip only contains relative paths so using simple string equality comparison easily leads to false negatives. Compare the full absolute path instead such that it doesn't matter anymore in which form the tests are passed. Note: - use realpath to resolve symlinks plus make the comparison case insensitive on windows - the test_file passed to run_one_test is not altered by this commit, such that when the user inputs relative paths the tests are also still displayed with relative paths - likewise the test_file_abspath is not modified because functions like run_micropython rely on it having forward slashes In practice this means that it used to be so that the only forms of running tests for which the skip_tests lists actually worked were: >python ./run-tests.py >python ./run-tests.py -d extmod whereas it now works consistently so also for these invocations, which in the end all point to the exact same path: >python ./run-tests.py -d ./extmod >python ./run-tests.py -d ../tests/extmod >python ./run-tests.py -d /full/path/to/tests/extmod These examples used to not skip any of the tests in the extmod/ directory thereby leading to test failures. Signed-off-by: stijn <stijn@ignitron.net>

Scan the --test-dirs argument for the main tests directory being passed and if so do the same thing as if running from within that main test directory. In practice this makes the following (which used to counterintuitively try and fail to run the .py files in the tests/ directory itself) >python micropython/tests/run-tests.py -d micropython/tests do the same thing as >cd micropython/tests >python ./run-tests.py which is logical and convenient. Signed-off-by: stijn <stijn@ignitron.net>

Signed-off-by: stijn <stijn@ignitron.net>

stinos · 2026-02-11T11:34:15Z

Can you elaborate on this ?

I presume he just means to not "accidentally" add a leading ./ to the domain arg!

Yes that's it. Don't know where it came from but I have the habit of always using leading ./ and that is hard to unlearn just for running micropython tests. Probably because of working in different shells and some of them requiring files in current directory needing that in some circumstances. Plus powershell completion/fuzzy search/... I'm using adds it anyway.

Is skip tests logic the only thing that could get confused by the different path syntax?

I think so, yes. There were some cmdline/ tests which relied on relative paths though so I fixed that.

I also added a commit which fixes another issue I've been eying for a while: the test output (i.e. what is displayed, goes in results.json etc) also used to be literally what was passed. But that would result in different display for the exact same file depending on how the path was passed.

Also added another commit which makes passing the tests/ directory itself via -d work as expected (or at least as I would expect). I'm only +0.5 on that one though, because it's some extra code but at th same time I'm not sure anyone ever does this. It is consistent on the other hand.

Issue description is updated with more examples.

Test file paths which get passed to the run_tests function can be absolute or relative and with or without leading slash in the latter case, depending on the arguments to run-tests.py, but since that path is used to: - display which tests run - record which tests ran in the results.json - craft the filename for the .exp/.out file for failed tests it is desirable to always use the same file path irregardless of how the user passed the path. In practice this means that all forms of running our own tests like: >python ./run-tests.py -i extmod >python ./run-tests.py -d extmod >python ./run-tests.py -d ./extmod >python ./run-tests.py -d ../tests/extmod >python ./run-tests.py -d /full/path/to/tests/extmod will now consistently all display the tests like pass extmod/time_time_ns.py FAIL extmod/some_failing_test.py and produce output files like results/extmod_some_failing_test.py.exp results/extmod_some_failing_test.py.out instead of displaying/using the exact path as passed. For external tests, meaning not in the tests/ directory, we also want to be consistent so there the choice was made to always use absolute paths. Signed-off-by: stijn <stijn@ignitron.net>

github-actions · 2026-02-11T12:05:46Z

Code size report:

Reference:  zephyr: Add support for RISC-V RV32IMC native code emitter. [9721955]
Comparison: tests/run-tests.py: Output consistent test file paths. [merge of 6fc41e1]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

stinos added 5 commits February 11, 2026 11:20

tools/codeformat.py: Use input files when formatting python code.

7d8eb32

Signed-off-by: stijn <stijn@ignitron.net>

tests/run-tests.py: Add an argument for showing which tests would run.

b6da62f

This is convenient when trying to figure out the correct values for --include/--exclude/--test-dirs/... arguments. Signed-off-by: stijn <stijn@ignitron.net>

tests/cmdline: Make tests succeed irregardless of invocation path.

b93fe9e

Signed-off-by: stijn <stijn@ignitron.net>

stinos force-pushed the runtestspaths branch from e1c1c6f to fbbe098 Compare February 11, 2026 11:34

stinos force-pushed the runtestspaths branch from fbbe098 to 6fc41e1 Compare February 11, 2026 11:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Some run-tests improvements#18803

Some run-tests improvements#18803
stinos wants to merge 6 commits intomicropython:masterfrom
stinos:runtestspaths

stinos commented Feb 10, 2026 •

edited

Loading

Uh oh!

codecov bot commented Feb 10, 2026 •

edited

Loading

Uh oh!

Josverl commented Feb 10, 2026

Uh oh!

Josverl commented Feb 10, 2026

Uh oh!

andrewleech commented Feb 10, 2026

Uh oh!

stinos commented Feb 11, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

stinos commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

codecov bot commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Josverl commented Feb 10, 2026

Uh oh!

Josverl commented Feb 10, 2026

Uh oh!

andrewleech commented Feb 10, 2026

Uh oh!

stinos commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stinos commented Feb 10, 2026 •

edited

Loading

codecov bot commented Feb 10, 2026 •

edited

Loading

stinos commented Feb 11, 2026 •

edited

Loading