[Analytics] Find tools to collect data from social media accounts
Open, HighPublic


Although Facebook's insights utilities are okay for what we want, Twitter, G+ and Mastodon do not have built-in tools to collect some basic information, like the evolution of the number of followers over time.

There used to be ThinkUp, but that project is closed.

In the first part of this task I would like us to make a list of solutions that can help us collect data from our social media accounts and store it in parseable files.

In the second part, I would like us to work out how to implement these solutions so the collection can be done automatically at regular intervals, preferably on a server that is always on.

paulb created this task.Jul 10 2017, 9:53 AM
James added a subscriber: James.Jul 10 2017, 9:41 PM

Well, I'm not sure about the 2nd part of your question. For the 1st, however, Google+ Communities can get basic information by linking their Community Page to a Google Analytics account. As for Twitter, most organizations use the tools built into whatever URL shortening service they use (i.e. bit.ly, for example). I use bit.ly, and there are both the ability to make custom domains (a likely desire in our case) and provides analytics via real-time and reporting. I do not use Mastadon, so I'll reserve comment there.

Thanks @James. Would you use bit.ly to get click through stats?

In other news, CLI tools we can run using cron/anacron to collect simple data (e.g. number of followers on twitter) would also be perfectly fine at this stage.

Since Piwik can already track if people come to our articles from social networks, I'm not sure if URL shorteners provide us with any additional information. They don't track things like number of followers or number of likes or reshares anyway.

I'd also not advocate for the use of URL shorteners in general, for both privacy and security reasons (people have to click on a link without seeing the URL first, which no security-aware user should ever do).

Linking our G+ community page to Google Analytics sounds very useful, though (since it's on Google+ anyway, Google already has the data so there would be no additional privacy concern).

paulb added a comment.Jul 11 2017, 4:53 PM

I agree that using URl shorteners is probably redundant. That was my question, whether there was anything else you coudl track with shorteners

Privacy may not be a great concern. Firstly because by using social media, you are already relinquishing a large portion of your privacy to Facebook, G+ and Twitter. Secondly, because in no way do we (KDE) need detailed information from our accounts. At the moment, for example, all I am interested in is the number of followers we have at each moment in time and which tweets gains more traction (re-tweets and likes).

Sure, it would be nice to have fine-grained stats like we have in Piwik, but at the moment we don't even have the coarsest of stats for things like Twitter, and that is bad.

James added a comment.Jul 11 2017, 5:39 PM

The use of URL-shorteners is mainly used for fitting a Tweet with a link into an appropriate character count. Keep in mind, I don't think anyone will mind a shortened Tweet coming from an official Twitter account with a custom domain (i.e. http://AaBbCcDd@kde.ly or similar).

This looks extremely promising for our needs: https://polrproject.org/ | Github link: https://github.com/cydrobolt/polr
"Polr is a quick, modern, and open-source link shortener. It allows you to host your own URL shortener, to brand your URLs, and to gain control over your data. It's also GPLv2+ licensed. " Comes with full api documentation, and we can host it ourselves.

I did some basic research, and Twitter comes with it's own analytics that you can use once signed up for if the above does not give us all we need data-wise. I assume the service is free for basic things as I did not see anything on pricing. I did confirm that a data-export function is included. See the FAQ here: https://support.twitter.com/articles/20171990#

Potentially some pretty cool stuff, especially Polr.

James added a comment.Jul 11 2017, 5:40 PM
In T6481#100646, @paulb wrote:

Thanks @James. Would you use bit.ly to get click through stats?

Yes you can.

skadinna renamed this task from Find tools to collect data from social media accounts to [Analytics] Find tools to collect data from social media accounts.Jul 28 2017, 12:38 PM
paulb added a comment.Aug 21 2017, 9:26 AM

As long as we don't have a solutions, I am collecting data by hand and putting it into a spreadsheet shared on share.kde.org.