Beta testing program
Open, Needs TriagePublic

Description

Goals

  • Users are invited into testing stuff
  • Users actually test nightly and beta versions
  • Developers get feedback from the testing, other than an occasional bug report

Solution: Krita’s welcome page will show new things to test, with links to version needed and a short questionnaire. One feature per week, so our users are not overwhelmed.

Users do not need to register into any kind of program and they can do it as ad-hoc as they want. However, we should keep track of their contributions and let the world know how they are helping. TODO: how exactly?

Example of a feature to test
[FEATURE] Gamut masking (Next;2019-08-20 and later)

Gamut masking is a methodical approach to color enabling you to explore and analyze color, plan color schemes and guide your color choices. Check out the user manual for more information: https://docs.krita.org/en/user_manual/gamut_masks.html

How to test
Set the artistic selector to your liking (please share your configuration or a screenshot) and create some art using gamut masking (for more information on the subject: Color Wheel Masking and Shapes of Color Schemes by James Gurney). You can either use a default mask or create a new one. Then please share your experience in the following short questionnaire.

  1. Do you use gamut masking often or uncommonly? Does the implementation support your workflow?
  2. Do the default masks provide a good starting point? Is there some fundamental mask missing? If yes, please provide more details on the missing masks.
  3. Is the UI clear or did you have trouble finding the right controls? Do the dockers have useful defaults?
  4. What do you like and what could be done better?
  5. Did you encounter any bugs? If so, please submit them to bugs.kde.org. Should you need guidance, you can consult our guide to reporting bugs (link).

Technical solution

1, feed similar to news, with a corresponding article on the web and a questionnaire on something like google forms

  • Maybe a bit ‘hacky’, but might be easy to implement, with no additional dependencies
  • No telemetry - can be turned on by default

2, integration with kuserfeedback

  • As this is tied to telemetry, users have to actively and consciously consent and be informed about the data collected
  • Allows targeted personalized surveys. We can run multiple surveys at a time targeted at different users.
  • Needs new server-side app. I have no estimate off traffic and stored data at this time.

We can start with solution 1, then see where it goes and if there is need for something like solution 2.

Personnel

Implementation

  • Developer
  • Solution 1: krita.org webmaster
  • Solution 2: sysadmin (telemetry server setup)

Running the beta program

  • Someone to select features for testing, create the surveys and analyze the data; ideally two people, so they can share the load, or stand-in for each other

I am thinking that for the 'feed' what we could just do is a monthly post with new stuff? The biggest problem we've had up till now with keeping people up with our development, is that there's noone with the time to write these articles. That's how the 'this week in krita' posts stopped happening.

rempt added a subscriber: rempt.EditedAug 3 2019, 2:11 PM

I am thinking that for the 'feed' what we could just do is a monthly post with new stuff? The biggest problem we've had up till now with keeping people up with our development, is that there's noone with the time to write these articles. That's how the 'this week in krita' posts stopped happening.

Oh, that's a good idea. We have an extra reason for doing this now, so I'd like to figure out a way to do this again. Combine it with regular calls for testing of features when they land, and we could make a big step in community engagement; the only thing is that we need to setup a simple way to give feedback. Maybe google forms, maybe there's a wordpress survey plugin?

Well, we'll need to figure out a way to formalize it, so making it part of the actual meeting(what would we like to see in this month's development log) will be absolutely necessary.

That said, bundling the changes does go against the idea to keep it simple and not overload the user. I am afraid however, that between the release and the manual and the release notes, noone will have the time to create a formal testing section, and we'll need to find a way to simplify this process.

If we're going to increase the amount of surveying we do, I would like to go away from google forms. This has been fine for previous surveys, but maybe we need to see if there's something on the KDE infrastructure we could use instead?

Concerning the surveys themselves, can we use https://survey.kde.org/?

Yes, exactly. That one's 'solved' then(well, we need to ask sysadmin for further help, but...).

rempt added a comment.Aug 3 2019, 2:43 PM

Maybe I should extend our news widget so it can show icons for different kinds of news...

Monthly development news is a good idea. I'm not sure that it will work for getting people to test, or that it will cover all the features. I thought that making it one feature/change at a time (we should hand pick the right one though) would make it seem more easy and quick and thus attract more testers.

As for the feature testing itself, we might be able to have a boilerplate questionnaire that would be quite easy to modify for the individual feature. The content does not need to be complicated. Also, we could work on it with the author when the feature is in merge request stage.

Then I'm thinking that maybe we should start with having the surveys linked inside the monthly post, like

This month in Krita

blabla

X has been working on feature Y and it got into a development build.
You can help us a great deal by getting the development build, testing this feature, and filling out the form. <link to form>

A has been working on feature B and it got into a development build.
You can help us a great deal by getting the development build, testing this feature, and filling out the form. <link to form>

And the first question on the form should be 'what is the git hash of the dev build you're using (where to find that). Remember, for this survey you need to test with a Krita:Next build from <date> or later.', so that we can check people are actually using the development build and not just pressing every link they come across.

rempt added a comment.Aug 3 2019, 3:05 PM
rempt added a comment.Aug 3 2019, 3:16 PM

Okay, that mail didn't work. I thought we might just put the survery link in the krita welcome page; that way people have to at least start the right version to do the survey.

Okay, that mail didn't work. I thought we might just put the survery link in the krita welcome page; that way people have to at least start the right version to do the survey.

We can have a list of surveys available for the particular version on the welcome page. We might also be able to integrate the LimeSurvey instance by some API.

However, we also need to tell the stable release users that there is a possibility to test something and how to get to the right build. For that, a news post would be good.

rempt added a comment.Aug 3 2019, 4:10 PM

That's what I meant: in the news post tell people about the new feature, and to download the stable nightly, which would have the survey link.

I have drafted a more detailed analysis, based on the ideas discussed in this task. The high level outline is there, but a lot of important detail is missing.

I have the work in progress document here: https://docs.google.com/document/d/1LjjWnZ7mv2UJ3mpSXSvxl0ulDau9l-GmkJA4Rq_9ClE/edit?usp=drivesdk