Running clang-format across all plasma (and potentially over repos)
Open, Needs TriagePublic

Description

Clang format

The problem

I hate review comments on whitespace. It's very time consuming, it wastes reviewers time, the committers time and most importantly it distracts from the actual content.

I see it happen with new contributors, and I have absolutely no doubt we lose potential future developers at this step.

It's a mundane task that a computer should be doing.

Rather than going through reviews line-by-line I would love to be able to just run one command and fix everything. This exists in a tool called clang-format. Other projects use this already.

I would like KDE to discuss having the same, I think it'll make reviews smoother, ease onboarding and free up valuable time for code.

A practical roll out

We would need to do the following:

  • settle on a clang-format configuration
  • run across all the relevant KDE codebase (with some form of human review)

Optionally:

  • Add to default arc lint file
  • have a git push hook on the server that rejects patches that fail the linter
  • have it as a step in the gitlab pipeline, though I don't think this is strictly necessary to do a roll out.

Even without the optional steps it still becomes easier to compress all review comments about whitespace into a single comment.

There's a great blog series [1] documenting MongoDB's port which is very thorough.
Blender [2] interestingly doesn't seem to have any server side hooks or verification.

The biggest challenges will be dealing with backports and rebasing existing branches in review

Testing

At a recent sprint we ran [3] on plasma-workspace and reviewed the diff.

There were a lot of changes, but nothing drastic.

There's a slight issue with multi-line q_property declarations.
At best i can set a flag to port them all to one line. Does someone have an idea on how to solve that?

The inevitable questions

"Can we check and format only the changed lines?"

Technically yes. The command becomes quite a bit more complex.

Pragmatically it's weird to tell someone to follow a style that isn't used elsewhere in that file.
Especially when that diff only contains a move, or you add one entry into an existing badly formatted switch.

I tried. The results were weird.

"it destroys git history!"

Knowing how to quickly select a parent revision in your chosen git ui is an essential part of using git annotation anyway, and the argument that it destroys history makes no sense, but in case someone wonders.

"git blame -w" acts as follows:

  • Re-ordering of includes does show up as a change
  • Other changes, including moving a brace to a new line, do not

If you do want to skip the revision in git blame completely, one can also use git hyper-blame which can skip entire commits
from a provided blacklist

something something forced behaviour

Realistically not everyone follows frameworks style. It would need to be opt in at a per repo level.

For the plasma-devel side of the mailing list, I would like to propose doing everything except:
oxygen, breeze, kwin(?)

does it cover qml?

No. I am unaware of a tool which does.

[1] https://engineering.mongodb.com/post/succeeding-with-clangformat-part-1-pitfalls-and-planning
[2] https://wiki.blender.org/wiki/Tools/ClangFormat#Migrating_Branches_and_Patches
[3] https://phabricator.kde.org/P434 place this in the top level of the source folder as .clang-format

ognarb added a subscriber: ognarb.Jul 11 2019, 3:16 PM

Will formatter ruin beautiful multi-line formatting into single line?

like here


or

?

Will formatter ruin beautiful multi-line formatting into single line?

like here


or

?

This will depends on the Style used. But clang format has a lot of configuration options: https://clang.llvm.org/docs/ClangFormatStyleOptions.html.

[4] https://code.qt.io/cgit/qt/qt5.git/tree/_clang-format
[5] https://github.com/qt-creator/qt-creator/tree/master/dist/clangformat#coding-rules-violated-by-clang-format


I tried my best to apply and enforce clang-format for my project, but I failed to do so in a reasonable time.
I've got an impression that clang-format configuration doesn't provide enough options to format code well enough, but maybe I should spend more hours :-).

akulichalexandr added a comment.EditedJul 11 2019, 4:14 PM

Will formatter ruin beautiful multi-line formatting into single line?

like here


or

?

This will depends on the Style used. But clang format has a lot of configuration options: https://clang.llvm.org/docs/ClangFormatStyleOptions.html.

Cool!
I want one member per line (I think such style is popular in KDE), what can I do? There are only true/false.

I think it would be nice to have ConstructorInitializerStyle with values 'one per line', 'all on one line', 'all on one line if fit', 'ignore' (keep as-is) and so on.

But we have only ConstructorInitializerAllOnOneLineOrOnePerLine.
It seems that MongoDB accepted that: https://github.com/mongodb/mongo/blob/master/etc/format_sample.cpp#L31

What a "well done" configurability...

Will formatter ruin beautiful multi-line formatting into single line?

In my config it'll still be multi line. Indentation of the lines can change.

But the best thing is to download the configuration above and try it.

Feel free to suggest any edits.

zzag added a subscriber: zzag.Jul 11 2019, 8:07 PM

Will formatter ruin beautiful multi-line formatting into single line?

It depends on whether ColumnLimit is set (as well as other relevant style options that control line breaking), but in general the best what you can do is just test the proposed .clang-format file for yourself and check whether multi-line formatting is ruined. ;-)

filipf added a subscriber: filipf.Jul 12 2019, 8:33 AM

In my config it'll still be multi line.

In fact, the suggested clang-format file does the same as mine.

Probably in your sample, the last enum value ends with a comma and that force clang-format to wrap it.

I can't find such a trick for the constructor initialisation list. With both configs (your and mine) clang-format is moving members around based on the line length.
I already played with ColumnLimit and all other options from the docs and various sources. IMO the result is disappointing and clang-format is not capable to do things right :-(

I'll be happy to use the tool when it gets more reasonable options. Current keys like ConstructorInitializerAllOnOneLineOrOnePerLine: true/false make no sense.

zzag added a comment.Jul 13 2019, 4:31 PM

I already played with ColumnLimit and all other options from the docs and various sources. IMO the result is disappointing and clang-format is not capable to do things right :-(

Have you tried disabling ColumnLimit (by setting it to 0)?

In T11214#191683, @zzag wrote:

I already played with ColumnLimit and all other options from the docs and various sources. IMO the result is disappointing and clang-format is not capable to do things right :-(

Have you tried disabling ColumnLimit (by setting it to 0)?

Yes

davidedmundson added a comment.EditedJul 14 2019, 4:39 PM

If I remove

BreakConstructorInitializers: BeforeColon

then I get initialisation unchanged, remaining at one per line:

+PowermanagementEngine::PowermanagementEngine(QObject *parent, const QVariantList &args)
+ : Plasma::DataEngine(parent, args)
+ , m_sources(basicSourceNames())

correctly

BreakConstructorInitializers: BeforeComma changes the output to an acceptable form.

Thanks, David. I don't have strong objections now.

At kphotoalbum, we just enabled clang-format for the code-base. So far, my experience was mostly boring (the good kind of boring).

Some pitfalls that I encountered so far:

  • IncludeBlocks: Regroup, can create false positives in commit-hooks -> https://bugs.llvm.org/show_bug.cgi?id=39327
  • Don't run clang-format (in-place) on symlinks (I hope this doesn't apply for most projects, but YMMV)

One other thing that we ran into in KPhotoAlbum is that apparently clang-format is not entirely stable in its results. I got slightly different results between 6.0.0, 7.0.1, and 7.1.0.
So when you create your .clang-format file you should do some testing to get consistent results across all developers.

Out of experience at work, were we "auto-reformat" stuff that way I can tell you that you can't avoid that. There are always some changes in different clang-format versions.
But normally that only affects << 1% of your sources.

For the proposed clang format.

In kate.git I now imported the own from Qt.
The only thing that I am at the moment not sure of is the column limit.
It is very short if you have long variable + function names.

We reviewed the results of the .clang-format on kate.git, we were not that happy with the re-flowing.
We tried a new variant that looks more promising, look at https://invent.kde.org/kde/kate/blob/master/.clang-format

Hmm, changes like these seem like a reduction in readability to me:

That is a un-avoidable trade off between reflow stuff and line length limit.

cullmann added a comment.EditedSep 11 2019, 3:05 PM

Same happens for large if clauses, btw. On the other side this is at least consistent.

The question is whether you really need the ColumnLimit - if you set a ColumnLimit of 0, clang-format won't reflow continuations like in Nate's first screenshot.
For Nate's second suggestion, BreakConstructorInitializers and BreakInheritanceList should produce more readable formatting.
I'm not sure which option can be used to fix the Enum formatting, but with ColumnLimit set to 0 at least the existing enums shouldn't be butchered, I guess...

For KPhotoAlbum, I've also set CommentPragmas to '^ (FALLTHROUGH|krazy:) ' to prevent comments with semantic meaning from being moved around.

From a robustness viewpoint it seems to me that SortIncludes should be enabled. Apart from uncovering bugs in header files it also helps with keeping the include list sorted. By also setting IncludeBlocks: Preserve you can still keep semantically grouped include blocks.

I just tried out the variant without column limit as attached to the other request on ktexteditor.git.
As thought you will get changes like that (with a >> 500 columns line as result)

  • return toCheckState(
  • currentStyle->foreground() == defaultStyle->foreground()
  • && currentStyle->background() == defaultStyle->background()
  • && currentStyle->selectedForeground() == defaultStyle->selectedForeground()
  • && currentStyle->selectedBackground() == defaultStyle->selectedBackground()
  • && currentStyle->fontBold() == defaultStyle->fontBold()
  • && currentStyle->fontItalic() == defaultStyle->fontItalic()
  • && currentStyle->fontUnderline() == defaultStyle->fontUnderline()
  • && currentStyle->fontStrikeOut() == defaultStyle->fontStrikeOut());

+ return toCheckState(
+ currentStyle->foreground() == defaultStyle->foreground() && currentStyle->background() == defaultStyle->background() && currentStyle->selectedForeground() == defaultStyle->selectedForeground() && currentStyle->selectedBackground() == defaultStyle->selectedBackground() && currentStyle->fontBold() == defaultStyle->fontBold() && currentStyle->fontItalic() == defaultStyle->fontItalic() && currentStyle->fontUnderline() == defaultStyle->fontUnderline() && currentStyle->fontStrikeOut() == defaultStyle->fontStrikeOut());

I think you should reconsider what is worse: sometimes a bit strange line breaking or such things ;=)
In AbsInt we tried it without, too, and did run in a lot of such cases.

sitter added a subscriber: sitter.Oct 3 2019, 12:37 PM

^ The diff for people not fluent in reverse-markdown ;)

-            return toCheckState(
-                       currentStyle->foreground() == defaultStyle->foreground()
-                       && currentStyle->background() == defaultStyle->background()
-                       && currentStyle->selectedForeground() == defaultStyle->selectedForeground()
-                       && currentStyle->selectedBackground() == defaultStyle->selectedBackground()
-                       && currentStyle->fontBold() == defaultStyle->fontBold()
-                       && currentStyle->fontItalic() == defaultStyle->fontItalic()
-                       && currentStyle->fontUnderline() == defaultStyle->fontUnderline()
-                       && currentStyle->fontStrikeOut() == defaultStyle->fontStrikeOut());
+                return toCheckState(
+                    currentStyle->foreground() == defaultStyle->foreground() && currentStyle->background() == defaultStyle->background() && currentStyle->selectedForeground() == defaultStyle->selectedForeground() && currentStyle->selectedBackground() == defaultStyle->selectedBackground() && currentStyle->fontBold() == defaultStyle->fontBold() && currentStyle->fontItalic() == defaultStyle->fontItalic() && currentStyle->fontUnderline() == defaultStyle->fontUnderline() && currentStyle->fontStrikeOut() == defaultStyle->fontStrikeOut());

Yeah, thanks ;=)

ktexteditor.git and kate.git use now btw. the file with the column limit.

If you find some consensus, we can sync again.

The file from Kate/KTextEditor seems to be spreading:

Git commit 8053b08d1806ad38a6631023f710f3ab8994af6e by Andreas Cord-Landwehr.
Committed on 06/10/2019 at 13:17.
Pushed by cordlandwehr into branch 'master'.

Use Kate's clang-format configuration

Only exception: sort include files, as one can use an empty line
to enforce separation of includes.

A +88 -0 .clang-format
A +6 -0 reformat.sh

https://commits.kde.org/artikulate/8053b08d1806ad38a6631023f710f3ab8994af6e

I synced the "sort include" change back to Kate/KTextEditor, that makes sense.

Rather than eventually duplicating this formatting definition file across dozens of repos, could we put a single copy if it in one place and have everything use it from there?

:=) If there is some consensus, this can be tried.
One issue with that is that the tooling wants to have it in the source tree.
We would e.g. add some ECM magic for that, that copies a file shipped with ECM to the source tree if we agree that the style is ok.

ognarb added a comment.Oct 6 2019, 5:16 PM

We could also enable the linter in Phabricator, similar to how this was done in the llvm phabricator instance: https://reviews.llvm.org/D49116

I would not spend time on Phabricator as we anyways will leave it behind not that far away ;=)

See D24568 for the draft of an ECM module that applies the Kate/KTextEditor/Artikulate style.
Is the same file as here, with sorting of includes, but with column limit.

davidedmundson closed this task as Resolved.Nov 7 2019, 2:58 PM
davidedmundson claimed this task.

Update on the plasma side.

The ECM module is included in every plasma project's CMakeLists.txt file. The result of running the script is not yet pushed.
It will be in this state for 10 more days for devs to raise final objections and or whitelist any parts of code if needed.

Then we'll push the formatted version.


Frameworks we should maybe bring this up at the relevant meeting next month.

10 days have passed. What's the status of this?

Objections were raised.

Plasma is now on pause :(

The original files here seemed to format things ok but the ones merged into frameworks caused really long lines. Mostly it's all about the column length thing above.

As noted in

https://phabricator.kde.org/D24568

  1. for the lambda stuff and the comments moving, there is a simple fix, we can merge, if wanted
  2. for the column wrapping, you need to try different widths and if that helps we can try to find some consensus, but only if somebody does that try and error a bit more, I pasted some examples in the other ticket.

I've just found this for QML files

https://github.com/Orange-OpenSource/qmljsreformatter

Will give it a try and report back

Maybe we can use the upstream formatter? It is available since Qt 5.15.

https://code.qt.io/cgit/qt/qtdeclarative.git/tree/tools/qmlformat

mart reopened this task as Open.Dec 14 2020, 12:12 PM
alex added a subscriber: alex.Dec 14 2020, 1:01 PM

Hmm, changes like these seem like a reduction in readability to me:

Can be fixed by adding // at the and, IMHO that is an even better tradeoff

Solved by adding a comma at the end.

One could manually didable clang-format for that bit, or use AlignConsecutiveAssignments (https://clang.llvm.org/docs/ClangFormatStyleOptions.html). But in a lot of places we don't use this, so I would argue that it does not make sense to enable it.

alex added a subscriber: mart.Dec 14 2020, 3:41 PM

Considering that there are several improvements to the clang-format config we can proceed. We briefly discussed this on the Plasma meeting. There it was discussed that Gitlab-CI is still in alpha version and can not be used for a ton of projects.
And alternative would be to use a commit hook.

My idea would be to proceed the following:

  • Add the clang-format targets where they don't already exist
  • Start reformatting the projects, I would prefer to start with smaller projects with few open MRs first. This wil of course require manual intervention in cases where clang-format would uglify the code.
  • Start formmatting the bigger projects like kio or plasma-workspace. To avoid much clashing with open MRs we could format individual components like tests/applets/runners. But then we could not have a single commit we could use as a reference for git blame

Also the CMake files are sometimes horribly inconsistent, maybe we could utilize https://github.com/cheshirekow/cmake_format, but that is rather low priority.

@mart @davidedmundson @nicolasfella Feel fee to comment 😄

Doing KIO and other frameworks requires a discussion on the frameworks devel mailing list.

I don't think we need to wait for frameworks to start deploying on plasma.

alex added a comment.Dec 27 2020, 10:35 AM

Two things I am currently not 100% happy with:

  • Same goes for some short functions, like getters that just return a member variable, currently they would get wrapped onto multiple lines. And IMO that looks bad and wastes a bunch of lines, an example where this is heavily used is yakuake/app/skin.h

I wish we could say that if the lambda/method fits on a single line it should not be modified, only if it is too long it should be wrapped.

Excluding some sections from the formatting will of course still be an option. What do others think about this?

I think consistency trumps all things.

I don't think we should start to exclude stuff, that just will lead to "yeah, I don't it for my 2 files, exclude it".

If you look at KTextEditor, I don't think it does horrible stuff.

If there are a few minor tweaks to be done for the default clang-format file, we can discuss that thought.

Btw., in AbsInt we use this file (or at least the base variant of it that I submitted initially) now since >> 5 years or so on a rather large code base and beside the not that nice collapsing of if (... || ... | ...) conditions, nothing that ugly happens.
(and it stands to reason if if-conditions that are more or less 10 lines long are a good idea anyways code wise)

alex added a comment.Dec 27 2020, 9:33 PM

If there are a few minor tweaks to be done for the default clang-format file, we can discuss that thought.

My idea was that we should settle for a config where most people can be happy with to avoid having to re-format entire projects with bigger changes ;)

But since we are pretty much happy with the current config we can utilize that.

As we have discussed in the plasma meeting a pre-commit hook would be very suitable to prevent unformatted code from being pushed. See https://invent.kde.org/frameworks/extra-cmake-modules/-/merge_requests/65 for a proposal.

alex added a comment.Dec 29 2020, 9:54 PM

But one more thing: clang-format can't take care of the parentheses, the KDE coding style say that one must set these. Even in if statements or for loops with one line. IMO it could make sense to run clang-tidy while we are reformatting the code.
Especially in old parts of the codebase that is done really inconsistent.

See https://stackoverflow.com/a/63324313

alex added a comment.Jan 16 2021, 1:01 PM

Maybe we can use the upstream formatter? It is available since Qt 5.15.

https://code.qt.io/cgit/qt/qtdeclarative.git/tree/tools/qmlformat

This removed the curly braces if the if statement has only one line . I do not see any option to change this behavior.

alex claimed this task.Jan 16 2021, 3:00 PM