benchdnn: --skip-impl does not work as expected for the --reorder driver

[Sqvid]

Summary

--skip-impl does not work as expected for the --reorder driver. It skips all implementations rather than just the matching one.

Version

main: f1af0d5

Environment

oneDNN includes hardware-specific optimizations and may behave
differently on depending on the compiler and build environment. Include
the following information to help reproduce the issue:

• OS version: Linux 6.14
• git hash: f1af0d5
• Platform: Seen on x64 and AArch64

Steps to reproduce

All impls skipped rather than only those matching "jit":

$ ./build/tests/benchdnn/benchdnn --reorder --skip-impl='jit' --allow-enum-tags-only=0 --sdt=f32 --ddt=f32 --stag=ab --dtag=ba 32x32
0:SKIPPED (Skip-impl option hit) (0 ms) __REPRO: --reorder --allow-enum-tags-only=false --sdt=f32 --ddt=f32 --stag=ab --dtag=ba --skip-impl=jit 32x32
tests:1 passed:0 skipped:1 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total: 0.00s; create_pd: 0.00s (9%); create_prim: 0.00s (0%); fill: 0.00s (0%); execute: 0.00s (0%); compute_ref: 0.00s (0%); compare: 0.00s (0%);

But manually commenting out all impls above simple_reorder in cpu/reorder/cpu_reorder_regular_f32_f32.cpp gives the following as expected:

$ ./build/tests/benchdnn/benchdnn --reorder --skip-impl='jit' --allow-enum-tags-only=0 --sdt=f32 --ddt=f32 --stag=ab --dtag=ba 32x32
0:PASSED (9 ms) __REPRO: --reorder --allow-enum-tags-only=false --sdt=f32 --ddt=f32 --stag=ab --dtag=ba --skip-impl=jit 32x32
============================================================
= Implementation statistics (--summary=no-impl to disable) =
============================================================
| simple:any : 1 (100%)                                    |
============================================================
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total: 0.01s; create_pd: 0.00s (1%); create_prim: 0.00s (0%); fill: 0.00s (18%); execute: 0.00s (0%); compute_ref: 0.00s (0%); compare: 0.01s (75%);

Observed behavior

--skip-impl returns early rather than searching to the end of the list.

Expected behavior

Only an implementation that matches the skip filter should be excluded from the impl search. I'd expect to see the same as what happens when I comment out implementations manually.

[Sqvid]

cc: @dzarukin @vpirogov

[dzarukin]

@Sqvid there's no public iterator interface for reorder primitive (as it has an outstanding API definition), thus, the behavior is expected and the only way to iterate over is to comment out implementations in the list (unfortunately).

[Sqvid]

Thanks for the explanation. Is the problem that reorder has several separate impl lists? (e.g, cpu_reorder_regular_T1_T2.cpp)

[dzarukin]

Thanks for the explanation. Is the problem that reorder has several separate impl lists? (e.g, cpu_reorder_regular_T1_T2.cpp)

Reorder is missing this interface supported which delivers the ability to pick an implementation of choice.