Relevant:

https://manual.limesurvey.org/Survey_settings < general settings
https://manual.limesurvey.org/Questions_-_introduction < questions intro.
https://manual.limesurvey.org/Question_types < questions possible.

We could ask for testing the bugs that were fixed in this report, but it's regressions we want to catch, so that's not that useful. Maybe we should compile a list of frequent regressions...

Things that I think would be useful are:

• OS {linux, windows 7, windows 8, windows 10, macos}
• GPU {intel, nvidia, amd}
• Memory

(Though these three could conceivably be replaced by asking people to add the krita.log file, having them as questions makes sorting and reporting easier)

• Did painting feel faster/slower
• Did moving layers feel faster/slower
• Did using the transform tool feel faster/slower
• Please try to create a new text object, edit the text, change font, save, load -- was this loaded correctly?
• After using the build for some time, have you experienced crashes?
• Free field to add any additional impressions or regressions found.
• Contact information if the tester would like us to contact them about their findings

I think we should start with a short description of the survey, what is expected of the participants and point to our reporting bugs docs page, should the participant encounter a bug while testing.

I agree with @rempt on the list of system information. At the very least we need to know the participant’s operating system and it’s version. Also, it might be interesting to know the hardware/tablet the participant used for the testing, if any (especially since we currently have bug 410797). The information does not need to be detailed, for most of them we can use single choice answers. I wouldn’t vote for having participants add krita.log, for the reasons mentioned and also because it might contain sensitive information.

For measuring conversion rate, we would also need to know how many people downloaded the beta, for which platforms. @scottpetrovic, can we get that information from the website? Do you have better ideas for measurement?

As for the things for testing, I do not think we should let the testers retest all our fixes with strict test cases. But it would be beneficial to have multiple optional steps, some of them asking about recent bugs/fixes, or areas we touched in the code (there are some interesting fixes in last two weeks, at least the freeze while painting and fixes for vectors). The last step would be to let the participant do what they usually do in Krita, and describe that workflow to us. At the end we might ask for some feedback on general feel of that version. What I describe here is basically what @rempt suggests in the second half of the post, just more structured.

Anonymize: yes.

I thought it would be nice to know who participated, so we can thank them publicly (if they wish so). If we wish to do it, we can still do it with anonymized survey: we would ask for a name/nick in the survey itself, together with a question, whether the participant does wish to be publicly named in the list of testers for this release with a big fat yes or no. (GDPR; participants have their rights). If the participant wants to stay anonymous, they can leave it blank/set to no.

Possible fixes to ask about:

• Possibly fix freeze on painting when using weird tablet drivers - https://invent.kde.org/kde/krita/commit/1a1ba081471bc90fe456609c2dbecc9af14f5cfb
• Fix crash when creating a bezier curve - https://invent.kde.org/kde/krita/commit/c025b251079158f2206c3a1d29b3068062310077
• Fix deadlocks in KoShapeManager caused by bezier curve fix - https://invent.kde.org/kde/krita/commit/c2d3c64b9be35b9ade4658f7b26e0a86f38e9a7c
• Fix mixed Zoom, Pan, Rotate on macOS - https://invent.kde.org/kde/krita/commit/56c77bb7f071127591738f16a69b3b720b83b97a

The platform really shouldn't be anything special we need to do. If they are downloading an appimage, we can be pretty confident they are on linux for example.

If you are wanting very specific details about their system along with their OS, it sounds like it needs to all be part of the survey. Ask for their system information, and give them the beta link right in the survey. Can that be done?

A side note for nightly builds and Jenkins. That Jenkins site has no web tracking set up, so we have no idea how many people are downloading nightly builds right now. The beta build needs to be stored on downloads.kde.org or files.kde.org for them to appear in Matomo, which I think we normally do anyways for beta builds.

I would like to start with one single survey for all platforms first.

So...

Basic Questions

What is your operating system
{linux, windows 7, windows 8, windows 10, macos]

What is the brand and model of your tablet?
[Single fill in line.]

How much ram do you have?
[free fill in...]

Can you copy-paste the OpenGL Info section from help->show system information for bugreports
[multiline text area]

Performance Questions

How does painting feel compared to the previous version?
Rate 1-5, with 3 being the same, 1 being slower, 5 being faster.

How does using the move tool compare to the previous version?
Rate 1-5, with 3 being the same, 1 being slower, 5 being faster.

How does the transform tool feel compared to the previous version?
Rate 1-5, with 3 being the same, 1 being slower, 5 being faster.

Specific Bugfix questions

<Explaination on how to get backtraces for windows... appimages cannot give backtraces, afaik?, Need more help with how to do that on MacOS>

Does using vector layers cause any new trouble?

• No
• Yes, [free line to fill in]

Please try to create a new text object, edit the text, change font, save, load -- was this loaded correctly?

• Yes
• No [free to fill in line]

After using the build for some time, have you experienced crashes?

• Yes
• No

After switching between applications, have you experienced crashes?

• yes
• no

(If on MacOS... there seems to be the option to ask conditional questions based on previous answers, so I'll try that)
Does pinch/zoom/rotate work for you?

• yes
• no [free space to fill in]

(I am not really sure what else we can ask here...)

Final page

This survey is anonymized, if you want to allow us to contact you, please leave behind contact information
[multiline text area]

Do you have any other thoughts you wish to add?
[multiline text area]

@woltherav Looks good. The only thing I might want to have clarified is what "previous version" means. I guess for this survey, it means how does this beta build compare to our current stable release

We could try to skip the strip step for the beta appimage builds, then backtraces should happen automatically. For macOS, the instructions are just too complicated for the average user, so we'd have to skip that. For the crash question, a follow-up asking whether the user has found out how to repeat the crash and what steps are needed to reproduce the crash would be useful.

Yes going the step of generating a backtrace would be complicated unless its bundled already, but if they can at least describe the steps to reproduce the bug that would be sufficient (at least on macOS)

It looks great :)

With the yes/no questions for bug fixes, is there a third state by which we know the participant did not test that one when analysing the answers?

Can you copy-paste the OpenGL Info section from help->show system information for bugreports

Let's remember that this part of Krita shows several OpenGL info logs so we need to make sure testers always take the *last* one. Looks obvious to us, but can be not that obvious to users, even those who want to participate in beta testing.

Here's some things I think we should encode in the URL:

KritaVersion
BuildAbi
BuildCPUArchitecture
CurrentCPUArchitecture
KernelVersion
ProductType (this is the OS)
ProductVersion (this is the OS version)
InstantPreviewEnabled
OpenGLRenderer
OpenGLVersion
OpenGLFormat
GPUVendor
TotalMemorySize
SwapLocation
NumberOfCores
HiDPI

InstantPreviewEnabled becomes InstantPreviewEnable, because the question code can't get that long.

OpenGLFilterMode, AngleRenderer, HiDPI, RealMemorySize is not in the link generator here: https://invent.kde.org/kde/krita/commit/d8b659f6f027df54b13fbdcf4709b73c233cf274

HiDPI is there -- arguments += "&HiDPI=" + QString(QCoreApplication::testAttribute(Qt::AA_EnableHighDpiScaling) ? "true" : "false");

I'll change the InstantPreview variable, the other three we can disregard -- I had already removed them from my comment, so this is the full list:

KritaVersion
BuildAbi
BuildCPUArchitecture
CurrentCPUArchitecture
KernelVersion
ProductType
ProductVersion
InstantPreviewEnable
OpenGLRenderer
OpenGLVersion
OpenGLFormat
GPUVendor
TotalMemorySize
SwapLocation
NumberOfCores
HiDPI