[Analytics] Find tools to collect data from social media accounts
Closed, ResolvedPublic

Description

Although Facebook's insights utilities are okay for what we want, Twitter, G+ and Mastodon do not have built-in tools to collect some basic information, like the evolution of the number of followers over time.

There used to be ThinkUp, but that project is closed.

In the first part of this task I would like us to make a list of solutions that can help us collect data from our social media accounts and store it in parseable files.

In the second part, I would like us to work out how to implement these solutions so the collection can be done automatically at regular intervals, preferably on a server that is always on.

paulb created this task.Jul 10 2017, 9:53 AM
James added a subscriber: James.Jul 10 2017, 9:41 PM

Well, I'm not sure about the 2nd part of your question. For the 1st, however, Google+ Communities can get basic information by linking their Community Page to a Google Analytics account. As for Twitter, most organizations use the tools built into whatever URL shortening service they use (i.e. bit.ly, for example). I use bit.ly, and there are both the ability to make custom domains (a likely desire in our case) and provides analytics via real-time and reporting. I do not use Mastadon, so I'll reserve comment there.

Thanks @James. Would you use bit.ly to get click through stats?

In other news, CLI tools we can run using cron/anacron to collect simple data (e.g. number of followers on twitter) would also be perfectly fine at this stage.

Since Piwik can already track if people come to our articles from social networks, I'm not sure if URL shorteners provide us with any additional information. They don't track things like number of followers or number of likes or reshares anyway.

I'd also not advocate for the use of URL shorteners in general, for both privacy and security reasons (people have to click on a link without seeing the URL first, which no security-aware user should ever do).

Linking our G+ community page to Google Analytics sounds very useful, though (since it's on Google+ anyway, Google already has the data so there would be no additional privacy concern).

paulb added a comment.Jul 11 2017, 4:53 PM

I agree that using URl shorteners is probably redundant. That was my question, whether there was anything else you coudl track with shorteners

Privacy may not be a great concern. Firstly because by using social media, you are already relinquishing a large portion of your privacy to Facebook, G+ and Twitter. Secondly, because in no way do we (KDE) need detailed information from our accounts. At the moment, for example, all I am interested in is the number of followers we have at each moment in time and which tweets gains more traction (re-tweets and likes).

Sure, it would be nice to have fine-grained stats like we have in Piwik, but at the moment we don't even have the coarsest of stats for things like Twitter, and that is bad.

James added a comment.Jul 11 2017, 5:39 PM

The use of URL-shorteners is mainly used for fitting a Tweet with a link into an appropriate character count. Keep in mind, I don't think anyone will mind a shortened Tweet coming from an official Twitter account with a custom domain (i.e. http://AaBbCcDd@kde.ly or similar).

This looks extremely promising for our needs: https://polrproject.org/ | Github link: https://github.com/cydrobolt/polr
"Polr is a quick, modern, and open-source link shortener. It allows you to host your own URL shortener, to brand your URLs, and to gain control over your data. It's also GPLv2+ licensed. " Comes with full api documentation, and we can host it ourselves.

I did some basic research, and Twitter comes with it's own analytics that you can use once signed up for if the above does not give us all we need data-wise. I assume the service is free for basic things as I did not see anything on pricing. I did confirm that a data-export function is included. See the FAQ here: https://support.twitter.com/articles/20171990#

Potentially some pretty cool stuff, especially Polr.

James added a comment.Jul 11 2017, 5:40 PM
In T6481#100646, @paulb wrote:

Thanks @James. Would you use bit.ly to get click through stats?

Yes you can.

skadinna renamed this task from Find tools to collect data from social media accounts to [Analytics] Find tools to collect data from social media accounts.Jul 28 2017, 12:38 PM
paulb added a comment.Aug 21 2017, 9:26 AM

As long as we don't have a solutions, I am collecting data by hand and putting it into a spreadsheet shared on share.kde.org.

paulb added a subscriber: dkardarakos.EditedJul 30 2018, 10:45 AM

Not so much a tool, but to evaluate the state of the community and contributors, I have talked to Ben Cooksley and currently talking to @dkardarakos about ways of mining the SVN and Git repositories to figure out how the number of contributors and contributions has evolved over time. This will be able to useful to see if there is a trend and whether it needs correcting.

What we need to proceed:

  • find a contributor that had access to the repositories as far back as possible to use their account to mine data (@apol?)
paulb added a subscriber: apol.Jul 30 2018, 10:47 AM
paulb added a comment.Aug 8 2018, 8:50 AM
In T6481#152969, @paulb wrote:

Not so much a tool, but to evaluate the state of the community and contributors, I have talked to Ben Cooksley and currently talking to @dkardarakos about ways of mining the SVN and Git repositories to figure out how the number of contributors and contributions has evolved over time. This will be able to useful to see if there is a trend and whether it needs correcting.

What we need to proceed:

  • find a contributor that had access to the repositories as far back as possible to use their account to mine data (@apol?)

@dkardarakos provided a pretty awesome dashboard to see this information and it goes all the way back to 1989 (!). It is currently hosted on my own server on the Internet:

https://quickfix.es:8983/solr/banana/#/dashboard/solr/KDE%20Totals?server=%2Fsolr%2F
https://quickfix.es:8983/solr/banana/#/dashboard/solr/KDE%20Projects?server=%2Fsolr%2F
https://quickfix.es:8983/solr/banana/#/dashboard/solr/KDE%20Translations?server=%2Fsolr%2F

Ideally, we should move these to a KDE-owned server at some point, but, for the time being, I am happy to host them.

Access requires a username and password, which is okay, because some of the information may be sensitive for KDE. If you need access. just ping me and I'll get to it.

Hey all, not sure if you were aware of this, but Piwik (the tool we use instead of Google Analytics for kde.org) is now called Matomo: https://matomo.org/

It might be a good idea to check if there are any new features or major changes that impact our use case (leaving this as a note to self, but anyone is free to check and report back :)).

paulb added a comment.Aug 12 2020, 8:38 AM

This has recently changed. Most social media sites now offer detailed downloads of insights. Closing this.

paulb closed this task as Resolved.Aug 12 2020, 8:38 AM
xyquadrat raised the priority of this task from High to Needs Triage.