argparse drops '|'s when the arguments are too long

[al-caveman]

BPO	31768
Nosy	@ericvsmith, @mlouielu, @al-caveman, @iritkatriel
Files	31768.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-10-11.21:49:25.596>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = "argparse drops '|'s when the arguments are too long"
updated_at = <Date 2021-12-10.16:33:55.067>
user = 'https://github.com/al-caveman'

bugs.python.org fields:

activity = <Date 2021-12-10.16:33:55.067>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-10-11.21:49:25.596>
creator = 'caveman'
dependencies = []
files = ['47214']
hgrepos = []
issue_num = 31768
keywords = []
message_count = 9.0
messages = ['304184', '304186', '304187', '304188', '304189', '304202', '304284', '304287', '408221']
nosy_count = 5.0
nosy_names = ['eric.smith', 'paul.j3', 'louielu', 'caveman', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31768'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• GH-75949: Fix argparse dropping '|' in mutually exclusive groups on line wrap #142312
• [3.14] GH-75949: Fix argparse dropping '|' in mutually exclusive groups on line wrap (GH-142312) #142347
• [3.13] GH-75949: Fix argparse dropping '|' in mutually exclusive groups on line wrap (GH-142312) #142348

[al-caveman]

if you execute the code below, mutually exclusive agrs are separated by '|' as expected. but if you uncomment the 1 line down there, then the args list will be too long, and as argparse tries to wrap the args around, it drops all '|'s which semantically destroys the intended syntax of the arguments.

import argparse

parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
group.add_argument("-x", metavar='X', type=str, help="the base", nargs='?')
group.add_argument("-y", metavar='Y', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
#group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')

args = parser.parse_args()

[al-caveman]

forgot to add: when you execute the code, pass the argument '-h' in order to have the script generate the help menu text.

[ericvsmith]

Removing versions 3.4 and 3.8.

Attaching a reproducing script. Run it with a parameter of the number of arguments to add. The behavior changes between 7 and 8, although I'm not sure either is wrong, just different. This is from Windows:

% python3 31768.py 7
usage: 31768.py [-h] [-v | -q | -x [X] | -y [Y] | Z | Z | Z | Z | Z | Z | Z]

positional arguments:
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
  -x [X]         the base
  -y [Y]         the exponent

% python3 31768.py 8
usage: 31768.py [-h] [-v] [-q] [-x [X]] [-y [Y]]
                [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z]

positional arguments:
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent
  Z              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
  -x [X]         the base
  -y [Y]         the exponent

[al-caveman]

When | is dropped, it means that the arguments/options are no longer mutually exclusive, which renders the resultant help menu incorrect.

[ericvsmith]

Good point.

[mlouielu]

the format of usage is do at argparse.py:296.

So how do we format this kind of situation,
to be like this, if the group also too long?

  usage: test.py [-h]
                 [-v | -q | -x [X] | -y [Y] | Z | Z | Z | Z | Z |
                  Z | Z | Z]

Also, another bug I think is, given two mutul group,
it will not show out the correct grouping:

usage: test.py [-h] [-v] [-e] [Z] [G] [Z] [G]

positional arguments:
  Z              the exponent
  G              the exponent
  Z              the exponent
  G              the exponent

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -e, --eeeeee

--- 2 mutul group ---

import sys
import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
g2 = parser.add_mutually_exclusive_group()
g2.add_argument("-e", "--eplog", action="store_true")
g2.add_argument("-g", "--quiet", action="store_true")

for i in range(int(sys.argv[1])):
    group.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')
    g2.add_argument("z", metavar='Z', type=str, help="the exponent", nargs='?')

parser.parse_args(['-h'])

[paulj3]

As documented in many other issues, the usage formatter is brittle. It formats the individual usages, joins them into a string. Then if too long to fit on one line it tries t split into actions, etc. This split produces an assertion error if there are 'wierd' characters in the names (e.g. #[]).

With mutually exclusive groups it gets even worse. The brackets and | are spliced into the original string, and then excess [] and spaces are removed. Once recent issue complained about its handling of nested groups (which are borderline wrong).

So I"m not surprised that a long group that spans a couple of lines gets messed up. It requires a major rewrite, and even then I there will be formats involving groups that fall through the cracks.

[paulj3]

I've changed your example a bit to clarify some points.

    parser.add_argument('n', type=int)
    group = parser.add_mutually_exclusive_group()
    group.add_argument("-v", "--verbose", action="store_true")
    group.add_argument("-q", "--quiet", action="store_true")
    group.add_argument("-x", metavar='X', type=str, help="the base", nargs='?')
    group.add_argument("-y", metavar='Y', type=str, help="the exponent", nargs='?')

    for i in range(int(sys.argv[1])):
         group.add_argument("z%s"%i, nargs='?')

With n=0, usage is:

usage: PROG [-h] [-v | -q | -x [X] | -y [Y]] n

Notice that the positional is placed last. That is, regardless of the definition order, 'usage' shows optionals first, positionals last.

With 2 'z':

usage: PROG [-h] [-v] [-q] [-x [X]] [-y [Y]] n [z0] [z1]

All the mutually exclusive markings have been dropped. That's because the component actions are not consecutive, having been split up by the reordering of positionals. Marking the group is done only if it can do so in a simple minded way. The group testing is not dependent on being able to format it. That's done by a different part of the code.

For 7 'z' it splits the line. Positionals are printed on a new line.

usage: PROG [-h] [-v] [-q] [-x [X]] [-y [Y]]
            n [z0] [z1] [z2] [z3] [z4] [z5] [z6]

I noticed this problem with formating group positional in multiline usage some time ago, https://bugs.python.org/issue10984#msg192954

Another thing to watch out for - multiple positionals in a group don't make sense. Only the first one can ever be filled. I don't see a formal check, but it doesn't make logical sense.

If I change the code so it produces multiple optionals:

usage: PROG [-h] [-v | -q | -x X | -y Y | --z0 [Z0] | --z1 [Z1] | --z2 [Z2] |
            --z3 [Z3] | --z4 [Z4]]
            n

the group markings are preserved across lines. Note again that the positional (n) is on its own line.

---

Here's another weird behavior, with just one positional in the group:

1314:~/mypy$ python3 argdev/issue31768.py 1 -v 3
usage: PROG [-h] [-v] [-q] [-x X] [-y Y] n [z0]
PROG: error: unrecognized arguments: 3

It complains about not recognizing the '3', rather than complaining about 'z0' not allow with '-v'. That's because the '?' has already been satisfied with [] when parsing the 'n' (see https://bugs.python.org/issue15112).

---

In sum, I think this is pathological case that doesn't need to be fixed.

Usage is ok if there's a long list of optionals. Usage (and parsing) with just one positional in the group is brittle. We shouldn't expect it work with many positionals in the group.

---

@louie, you 2nd example interweaves the definitions of two groups. Except for the optionals/positions reordering that I mentioned above, usage does not reorder arguments. Mutually exclusive groups are marked only if they are contiguous.

https://bugs.python.org/issue10984 has a patch that can format groups even if the actions don't occur in the right order. It isn't a trivial fix.

[iritkatriel]

In 3.11 the problem is still there, but the cutoff to make Eric's script show it is now 26 rather than 8:

% ./python.exe tt.py 25
usage: tt.py [-h] [-v | -q | -x [X] | -y [Y] | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z | Z]

% ./python.exe tt.py 26
usage: tt.py [-h] [-v] [-q] [-x [X]] [-y [Y]]
             [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z] [Z]