Setting the 2024 stable style

[hauntsaninja]

See issue for 2023 here: #3407

Here's what this issue is:

• I've listed all the features we have enabled under --preview
• ...in descending order of how controversial I think they are
• I've also drawn a dividing line; past "uncontroversial" is where I feel we don't need to think twice about it
• I've listed the number of digits in the diff-shade lines changed count and linked original PR
• I tried to list any open issues next to the relevant preview feature, but it's possible I missed something
  • Issues tagged with preview I haven't yet associated with a feature: Preview style: Preserve empty lines between async def and async for #3614 and string_processing: makes it impossible to use a comment to ignore a static type violation #3802 (sorted these out–Jelle)

Here's the current call to action:

• Any comments / opinions / vibes on these features is very welcome
• Please test your codebase with --preview and let us know your thoughts!
• If you have a specific problem with a preview feature, especially something that feels like an implementation issue, it's best to open a separate issue (that we can then link to from here)

For many of the changes, diff-shades logs have expired. We should find a way to keep these around for longer!

It's currently November, so there's still time to make more uncontroversial changes and hopefuly iron out some issues in the not uncontroversial ones.

Not uncontroversial

string_processing

This one is controversial, vast majority of preview style issues are related to string processing
Open issues I believe are caused by string_processing: #2334, #2314, #3466, #3497, #3409, #3424, #3428, #3210, #3410, #3747, #3665, #3663, #2553, #3855, #3802

hug_parens_with_braces_and_square_brackets (#3964, #3992)

Causes O(10000) changes, although counting lines changed is unfavourable to this change.
A very substantial change in style for Black. We can't revert once it's in stable.
Outstanding crash: #4036

parenthesize_conditional_expressions (#2278)

Causes O(1000) changes. Need to look at diff
Has unfortunate interactions with comments: #3555

wrap_long_dict_values_in_parens (#3440)

Causes O(100) changes. Need to look at diff
Outstanding crash: #3758
Report of worse splitting: #3452
Unnecessary nesting: #4129
Removes parentheses: #4158

multiline_string_handling (#1879)

Causes O(1000) changes. Need to look at diff
Maybe has some synergy with hug_parens_with_braces_and_square_brackets.
Unfortunate white space preservation: #4159

dummy_implementations (#3796)

I can't find up-to-date diff-shades, but maybe O(1000) changes
Requires a change in flake8 config: #3887
Can result in an extra line, but no change expected to be made: #3872

prefer_splitting_right_hand_side_of_assignments (#3368)

Causes O(100) changes.

hex_codes_in_unicode_sequences (#2916)

Causes O(100) changes
Open issue whether lowercase or uppercase is better #3568

allow_empty_first_line_before_new_block_or_comment (#3967)

Causes O(1) changes.
This change itself is fine, but it's related to a controversial and churn causing change in 2023 style. I think we should probably revert that 2023 change; I posted a longer issue about this over here #4043

parenthesize_long_type_hints (#3899)

Causes O(100) changes

module_docstring_newlines (#3932)

Causes O(100) changes

Uncontroversial

respect_magic_trailing_comma_in_return_type (#3916)

O(10) changes.
Shouldn't affect existing formatting too much, unless there's a buggy trailing comma
All changes in diff-shades are incidental, but are good and what you'd expect Black to do

no_blank_line_before_class_docstring (#3692)

O(100) changes. Makes consistent with function behaviour
Blank lines have been a little controversial, but I think this should be totally fine

wrap_multiple_context_managers_in_parens (#3489)

O(100) changes. Naively, looks great to me

fix_power_op_line_length (#3942)

O(10) changes. A bug fix

add_trailing_comma_consistently (#3393)

O(1) changes. A bug fix

skip_magic_trailing_comma_in_subscript (#3209)

O(1) changes. A bug fix, affects a non-default option

blank_line_after_nested_stub_class (#3564)

Only affects stubs, adds some blank lines in an uncommon case

blank_line_between_nested_and_def_stub_file (#3862)

Only affects stubs, adds some blank lines in even less common case

accept_raw_docstrings (#3947)

O(1) changes. Just a bug fix

long_case_block_line_splitting (#4024)

0 changes. Seems strictly good

walrus_subscript (#3823)

0 changes. Straightforward style change

improved_async_statements_handling (#3609)

0 changes. More consistent

single_line_format_skip_with_multiple_comments (#3959)

0 changes. Makes life better

[hauntsaninja]

My vibes:

• I feel very comfortable with shipping dummy_implementations and lower
• I think parenthesize_conditional_expressions / wrap_long_dict_values_in_parens / multiline_string_handling are also all good, but I want to look over diff and address the existing issues
• I feel we shouldn't ship string_processing in its current state
• I'm fairly wary of hug_parens_with_braces_and_square_brackets. It's a big change to Black's style! I'm very interested in hearing from folks on this one; if we don't feel like we get much feedback on it we should strongly consider keeping in --preview
• I'll evaluate some more on large work codebase and post any findings

[AlexWaygood]

My ¢2 as a Black user:

1. I generally agree with @hauntsaninja's vibes/analysis overall!

2. I ran Black with --preview on a few projects, and in those projects, I'm sad to say that I dislike most of the string-formatting-related changes the preview style make.

3. I really, really like hug_parens_with_braces_and_square_brackets and hope it gets in! One of my biggest gripes with black is how annoying its formatting of multiline frozensets etc. has been in the past, and this addresses that. The one thing I'd say is that it surprised me that the preview style does this:

   -    return AnnotationStats(
   -        *[random.randint(0, 1000) for _ in AnnotationStats.__annotations__]
   -    )
   +    return AnnotationStats(*[
   +        random.randint(0, 1000) for _ in AnnotationStats.__annotations__
   +    ])

   I prefer the non-preview style there -- I believe this is basically the same thing @JelleZijlstra's is saying in Fix #4011: Handle more huggable immediately nested parens/brackets. #4012 (comment)

[JelleZijlstra]

Thanks, this is great!

We should try to make a 24.1a1 soon that implements a draft stable style, so we can get more feedback.

I haven't thought too hard about each change, but a few thoughts:

• It might be time to pull the plug on string_processing, since it's been years and the feature is still buggy. Possibly we can keep a more limited version, e.g. only merging adjacent string literals that are on the same line.
• I like the hug_parens change personally but I appreciate it's going to be controversial. Especially since it went into the preview style pretty late in the year, it might be better to hold off on it for now and gather more feedback.

[yilei]

On string_processing:

We have been using it internally (Google) since the beginning of this year and it has been working well for us. So I'm motivated to keep this and help graduate it to the stable style (too late for the 2024 stable style though).

Possibly we can keep a more limited version, e.g. only merging adjacent string literals that are on the same line.

This is a good strategy, I think we can split string_processing into a few separate style changes. Moving all of them at once is hard to reason about and is too disruptive.

[tungol]

I tested preview on a code base of mine, and it mostly looked good. The only change I was really surprised by was from string processing; the same issue as #3663 and #3210 . A little testing showed that it's from the string_paren_strip transformer. I like the idea of that part of string_processing, but that's a pretty ugly downside. It seems like it's stripping the parentheses and then re-inserting them at a worse place, which is consistent behavior with non-preview style if I manually remove the parentheses first. Not strictly a problem with string_paren_strip, I guess, but not great.

For hug_parens, I like it in theory but it does feel less consistent than the current behavior, at least in some cases. I'm not totally sure about it. I can understand why you'd compare it and multiline_string_handling, but I don't see any issues with the latter - that one looks great to me.

I think that parenthesize_conditional_expressions is a big improvement for conditionals in the original context of function arguments, but I don't think I like what it's doing in other contexts.

[JelleZijlstra]

I started putting together a PR to prepare an alpha for the 2024 stable style, and here's a few thoughts:

• hex_codes_in_unicode_sequences: I think there are good reasons to prefer lowercase. I would propose to keep this out of the 2024 stable style and make the 2024 preview style use lowercase.
• allow_form_feeds was just added (Permit standalone form feed characters at the module level #4021) as another preview feature. It should be low-impact, so I would favor adding it to the new stable style.
• I wouldn't be comfortable shipping parenthesize_conditional_expressions and wrap_long_dict_values_in_parens with the open issues about them. Nevertheless, I'll include them initially so we can gather more feedback. It's better if we put something in the alpha and then walk it back, instead of adding a new style change after the alpha.
• dummy_implementation is surprisingly disruptive in our test suite, but that's atypical code.
• The test suite found a new crash with parenthesize_long_type_hints: Crash with preview style + line length 1 #4062
• Reported suboptimal formatting with dummy_implementation: Preview style: Don't collapse dummy implementation with comment #4063

[DanielNoord]

Is there a chance of module_docstring_newlines making it in? As far as I can see there has only been support for the feature since it was first requested in 2020 (#1872) and I haven't seen issues about it being opened since it was added to preview.

If it means anything it was also quickly picked up by ruff in astral-sh/ruff#7995 without any discussion about it being potentially controversial.

It seems to me to be controversial as it would create a lot of diffs across projects, but doesn't change styling in any way that users would say "hey, what's black doing there". I would really like to see this change make it in to standardize these newlines across the projects I work on (and which don't want to run with preview style).

[JelleZijlstra]

@DanielNoord yes, that one is expected to go in (unless either lots of people speak up and say they don't like it, or we discover bugs caused by it).

[MichaReiser]

I'm still writing it up but we track Ruff's preview style compatibility in astral-sh/ruff#8678
Ruff's stable style already includes some of the mentioned uncontroversial preview styles.

Regarding multiline_string_handling. Ruff stable already formats

MULTILINE = """
foobar
""".replace("\n", "")

and we've generally received positive feedback. I haven't yet come around to implement the special casing inside call expressions. So I can't say how people feel about that.

I personally like hug_parens_with_braces_and_square_brackets that that's probably because I'm biased from Prettier ;)

Ruff formats long dict values slightly different from black (wrap_long_dict_values_in_parens). Parenthesizing the values would align the behavior. Although it sets a new precedence of removing parentheses from sub-expression, something Black so far very explicitly avoids (in comparison to Prettier that normalises all parentheses). There's a similar issue around slice indexes that may benefit from parenthesizing as well astral-sh/ruff#7316

I'll share more feedback when I had time to take a closer look at the changes and thanks for drafting this up, it helps to understand how Black's styleguide will change.