Create Task
Maniphest T11388

torrent generator
Closed, ResolvedPublic
Actions

Description

https://github.com/krisives/metalink-to-torrent

Seeing as the solution is based on metalinks and metalinks only get populated as mirrors pick up the file we may have to have a delay or generate the torrent multiple times.

Possibly a good approach would be:

  • publish iso
  • trigger secondary job
  • generate torrent from initial metalink (which only has the master server)
  • re-generate the torrent after 3 and again after 6 hours to pick up new servers

Torrent hashes are based on their info blob and so long as that doesn't change the torrent should be considered "the same". To inject new mirror seeds we do not need to change the info blob.

Open issues:

  • what to do about gpg signatures? should probably also add them to the torrent
  • ktorrent kaput https://bugs.kde.org/show_bug.cgi?id=382375 -> D23110 & D23111
  • in light of ktorrent being bust, should we maybe run a seed? it would kinda defeat the purpose of load balancing through mirrors though

Todo:

  • write rigging to wrap around generator
    • sftp locate iso
    • make expected meta4 and torrent url
    • download meta4 and torrent (dont fail on latter)
    • generate
    • push new torrent file
  • extend generator to update a torrent if it already exists
  • add job that runs every 3 hours?
    • possibly track remote list, if it didn't change we don't need to refresh the torrent

Related Objects

sitter created this task.Aug 12 2019, 1:12 AM
sitter triaged this task as Low priority.
sitter updated the task description. (Show Details)Aug 12 2019, 10:26 AM
sitter moved this task from Blocked to Ready To Do on the Neon board.
sitter updated the task description. (Show Details)
kives added a subscriber: kives.Aug 12 2019, 10:43 AM

in light of ktorrent being bust, should we maybe run a seed? it would kinda defeat the purpose of load balancing through mirrors though

In my opinion it moves the logic to the clients to do the load balancing. It's unlikely that a client will actually download from all of the mirrors at once but rather will connect to them and use the ones that are the fastest. From the perspective of most torrent clients, the mirrors are just another peer, and the normal bandwidth optimization strategies are used. It should still distribute the load to many mirrors.

sitter updated the task description. (Show Details)Aug 12 2019, 12:13 PM
sitter added a comment.Aug 12 2019, 12:21 PM
In T11388#195179, @kives wrote:

in light of ktorrent being bust, should we maybe run a seed? it would kinda defeat the purpose of load balancing through mirrors though

In my opinion it moves the logic to the clients to do the load balancing. It's unlikely that a client will actually download from all of the mirrors at once but rather will connect to them and use the ones that are the fastest. From the perspective of most torrent clients, the mirrors are just another peer, and the normal bandwidth optimization strategies are used. It should still distribute the load to many mirrors.

Sure, not with ktorrent though because webseeds are broken there. So the options are we run a torrent client to always provide a seed regardless of webseeds working, or we accept that ktorrent is broken and the torrent won't work. In the former scenario our seed may in fact end up the being the only one for considerable lengths of time thus not being balanced.

sitter moved this task from Ready To Do to Doing on the Neon board.Aug 12 2019, 1:22 PM
ngraham added a subscriber: ngraham.Aug 12 2019, 1:28 PM
ngraham added a subscriber: Neon.
kives added a comment.Aug 12 2019, 1:41 PM

I'm looking at the KTorrent bug and trying to fix it. I've got a working setup right now to build and debug libKF5Torrent.

sitter added a comment.Aug 12 2019, 1:52 PM

There's already fixes up for review D23110 & D23111 ;)

kives added a comment.Aug 12 2019, 11:32 PM

Wow really great work I just got a chance to test the KTorrent changes they work awesome!

sitter closed this task as Resolved.Aug 13 2019, 2:11 PM
sitter claimed this task.

torrents are now generated when isos are built and then get their mirror lists updated every 3 hours. thanks again for the generator code @kives!

including gpg signature idea was discarded because webseed paths are constructed from the name and the individual files for multi file torrents.
i.e.

name: 'foo'
files: [ { path: ['bar.iso'] } ]

results in webseed path construction http://.../foo/bar.iso

trouble is name also doubles as "hint" for a directory name. so we'd hint a useless directory name such as 20190811-1110 for the clients to put the files in, that name is of course uniquely garbage because it's useless to user as far as identifying what it is. there may be some way around that by using an empty name extend the path of the files but I didn't want to spend any more time on it. the sigs are small enough that users can just download it via the mirrors even when they are slow.

kives added a comment.Aug 14 2019, 4:00 AM

Great work. Really impressed by the turnaround time on this and the fixes to KTorrent, those are awesome.

I noticed when I click on the .torrent file from files.kde.org sometimes it redirects to a 404, I assume some mirrors haven't updated yet?

Also, how are users going to find the .torrent file from the neon.kde.org/download page in the future?

sitter added a comment.Aug 14 2019, 9:48 AM
In T11388#195423, @kives wrote:

Great work. Really impressed by the turnaround time on this and the fixes to KTorrent, those are awesome.

I noticed when I click on the .torrent file from files.kde.org sometimes it redirects to a 404, I assume some mirrors haven't updated yet?

Might have to do with the mirrors falling over, our sysadmins have disabled redirects now, so chances are this is solved. If not, please make note of the time should you notice it again.

Also, how are users going to find the .torrent file from the neon.kde.org/download page in the future?

We've decided against advertising this widely until ktorrent is fixed to prevent people from having a bad experience with ktorrent. A ktorrent release is scheduled for september.

kives added a comment.Aug 19 2019, 11:47 PM

Hey I noticed the torrent didn't generate anymore:

https://files.kde.org/neon/images/user/current/

• fusitter added a subscriber: • fusitter.Aug 29 2019, 3:58 PM

[spam comment removed by sysadmin]

kives added a comment.Aug 29 2019, 5:10 PM

@sitter Any idea what is going on here with this other user? I have no idea who he is or why he is upset.

sitter added a comment.Aug 30 2019, 1:22 PM

@kives spam.

I think I've got a handle on what was wrong with torrents meanwhile. The problem was that the piece calculation only gets queued in mirror brain, so right after ISO upload they aren't available. There's now fallback piece calculation in place to ensure the meta4 file matches the expectations vis a vis sha1 sums.

kives added a comment.Aug 30 2019, 8:21 PM

@sitter Thanks for the hard work. Does this mean we are waiting for the 5.1.2 release of KTorrent?

sitter added a comment.Sep 2 2019, 1:56 PM

Yep. It should be out this week, so we likely are good to promote torrents next week or the week after so distros had a chance to adopt it.

kives added a comment.Sep 3 2019, 7:14 AM

Thanks @sitter . I just got the 5.1.2 update in KDE Neon user updates today and it worked like a charm. Thanks again for all the amazing work. When the time comes we should do a post on /r/kde about it too!