Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 filesystem(4 GB)
ClosedPublic

Authored by shubham on Oct 16 2018, 1:53 PM.

Details

Summary

BUG: 198772

Test Plan
  1. Prepare copy/move job.
  2. Copy/move the contents to external USB drive/ Pen drive formatted with FAT32

Result: Inline error message appears.

Diff Detail

Repository
R241 KIO
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
There are a very large number of changes, so older changes are hidden. Show Older Changes
ngraham requested changes to this revision.EditedOct 16 2018, 11:01 PM

Thanks for this patch! There are a few things:

  1. Does this actually work? is info.bytesTotal() the largest size of a file that's permitted on the destination filesystem? If it is, then we need to check every file in the source, not the total size.
  2. Let's check for this condition before doing a check for whether there's enough free space on the disk, or else it will report a less-useful error message first (people could go try to make some space, only to discover that the filesystem doesn't even support copying a file that big)
  3. ERR_DISK_FULL will not display an accurate error message. Perhaps we should specifically check for FAT32 and define a new type of error message in src/core/job_error.cpp to handle this. Something like ERR_TOO_LARGE_FOR_FAT32 that has an appropriate error message; something like "The file <file> cannot be transferred because it is greater than 4 Gb in size, and the destination's filesystem does not support files that large."
  4. There's a TODO in this file about implementing this; please remove it.
This revision now requires changes to proceed.Oct 16 2018, 11:01 PM
shubham retitled this revision from Warn user before copy/move job if required space exceeds the maximum volume size of a File System to Warn user before copy/move job if the file size exceeds the maximum possible file size of in a File System.Oct 17 2018, 3:36 AM
shubham retitled this revision from Warn user before copy/move job if the file size exceeds the maximum possible file size of in a File System to Warn user before copy/move job if the file size exceeds the maximum possible file size in a File System.

Is there any predefined function which returns the file size limit for a given file system?

shubham updated this revision to Diff 43890.Oct 18 2018, 6:29 PM
shubham retitled this revision from Warn user before copy/move job if the file size exceeds the maximum possible file size in a File System to Warn user before copy/move job if the file size exceeds the maximum possible file size(4 GB) in FAT32 file system.
shubham edited the test plan for this revision. (Show Details)

Maybe we could cache the value of KFileSystemType::fileSystemType() somewhere so we don't need to calculate it again for every file, since it's the same for every one.

src/core/job_error.cpp
249

it's -> its

shubham updated this revision to Diff 43894.EditedOct 18 2018, 7:05 PM

Cache file system type instead of creating it every time

shubham marked an inline comment as done.Oct 18 2018, 7:06 PM
ngraham added a comment.EditedOct 18 2018, 7:35 PM

Thanks, this is looking better. I will test it out later.

It's not a huge deal, but 4294967296 plus a comment saying /* 4Gb */ or something might be clearer and more readable than qPow(4*M_E, 9). Then you could avoid adding the <QtMath> import too.

shubham updated this revision to Diff 43895.Oct 18 2018, 7:55 PM
shubham retitled this revision from Warn user before copy/move job if the file size exceeds the maximum possible file size(4 GB) in FAT32 file system to Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 file system(4 GB) .

Thank you for indulging me 😊

ngraham added inline comments.Oct 18 2018, 8:01 PM
src/core/global.h
249

Please add a comment with @since to this

src/core/job_error.cpp
253

Unrelated and incorrect change: https is correct here, so please revert.

bruns added a subscriber: bruns.Oct 18 2018, 8:47 PM
bruns added inline comments.
src/core/copyjob.cpp
1622

why not 1 << 32 - 1 ?
Also note the - 1, 2 ^ 32 is 0, as FAT32 uses a uint32_t for the file size.

src/core/job_error.cpp
249

%1 with quotes please, as it can contain whitespace, so it would be hard to spot where the filename ends.
I would also prefer a sentence where the reason is given first, e.g.
The destination filesystem only supports files up to 4GB. "%1" is to large an can not be transferred.

ngraham added inline comments.Oct 18 2018, 8:58 PM
src/core/job_error.cpp
249

None of the other error messages use quotes around the filenames. What's most semantically correct would be to change i18n to xi18n and wrap the filename/path token in <filename></filename> tags (this is exactly what they're for). We can do that in this patch, and then change it for all the other error messages (as appropriate) in another patch.

Also, in terms of wording, the HIG recommends putting the most important part of the sentence first, which in this case would be the error message; the explanation would go second. See https://hig.kde.org/style/writing/wording.html. So the current sentence may be a bit long, but it's structurally correct as-is.

cfeck added a subscriber: cfeck.Oct 18 2018, 9:03 PM
cfeck added inline comments.
src/core/copyjob.cpp
1622

Does this even work on a -m32 system?

bruns added inline comments.Oct 18 2018, 9:19 PM
src/core/job_error.cpp
248

Should also be handled in KIO::rawErrorDetail(...), same file below.

249

"Could not transfer %1 because it is to large. The destination filesystem only supports files up to 4GB. is shorter ...

The second part may even be removed alltogether, see KIO::rawErrorDetail, https://github.com/KDE/kio/blob/master/src/core/job_error.cpp#L311

bruns added inline comments.Oct 18 2018, 9:24 PM
src/core/copyjob.cpp
1622

(1ul << 32) -1 then ...

Tested this out and couldn't actually make it work the way I asked--with the FAT32 file size limit checked first. I think I see why: there are two total size > available space? checks that output ERR_DISK_FULL on error, and both of them execute first. The FAT32 file size limit check should be before the first one, probably near // Stat the next src url.

shubham updated this revision to Diff 43930.Oct 19 2018, 3:23 PM
shubham marked 5 inline comments as done.

File size limit check for a given file system before (m_totalSize > m_freeSpace) check

src/core/job_error.cpp
253

That was my previous commit where I changed http -> https. Don't know why it affected my local branch.

1ul << 32) -1 needs a comment explaining what it means, for people like me with puny brains. :)

There are two checks in this file that result in an ERR_DISK_FULL if there is not enough space. The FAT32 check is now before the second one, but not the first one, so the first one still will still look for adequate space before checking to see whether or not the copy is even possible due to FAT32 file size restrictions.

@ngraham No, please recheck, the check is before the first m_totalSize > m_freeSpace check

shubham updated this revision to Diff 43938.Oct 19 2018, 4:35 PM
bruns added inline comments.Oct 19 2018, 5:11 PM
src/core/job_error.cpp
1078

double "the"
drop the "`s"

1079

"Format with"
"filesystem type" or just "filesystem"
either "a file" or better "files"

@bruns Apart from these, does the code seems sane to you?

Can I get a review?

bruns added a comment.Nov 1 2018, 6:25 PM

Please update the diff.

src/core/job_error.cpp
1079

"transeferred"

bruns requested changes to this revision.Nov 1 2018, 6:25 PM
This revision now requires changes to proceed.Nov 1 2018, 6:25 PM
shubham updated this revision to Diff 44688.Nov 2 2018, 3:54 AM

"Format with destination drive to a filesystem type which" -> "Reformat the destination drive to use a filesystem that"

shubham updated this revision to Diff 44710.Nov 2 2018, 4:50 PM
shubham retitled this revision from Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 file system(4 GB) to Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 filesystem(4 GB) .

Uh-oh, I think the latest update to this diff isn't quite correct. I see unrelated changes that you didn't make being marked as additions.

You might try using arc in the future. It makes this process much simpler. See https://community.kde.org/Infrastructure/Phabricator#Using_Arcanist_to_post_patches

shubham updated this revision to Diff 44712.Nov 2 2018, 5:24 PM

Uh-oh, now the patch doesn't apply for me anymore:

$ arc patch D16249
 INFO  Base commit is not in local repository; trying to fetch.
Created and checked out branch arcpatch-D16249.
Checking patch src/core/job_error.cpp...
error: while searching for:
    case KIO::ERR_CANNOT_CREATE_SLAVE:
        result = i18n("Unable to create io-slave. %1", errorText);
        break;
    default:
        result = i18n("Unknown error code %1\n%2\nPlease send a full bug report at https://bugs.kde.org.", errorCode, errorText);
        break;
    }


error: patch failed: src/core/job_error.cpp:245
Hunk #2 succeeded at 1068 (offset -4 lines).
Checking patch src/core/global.h...
Checking patch src/core/copyjob.cpp...
Applying patch src/core/job_error.cpp with 1 reject...
Rejected hunk #1.
Hunk #2 applied cleanly.
Applied patch src/core/global.h cleanly.
Applied patch src/core/copyjob.cpp cleanly.

 Patch Failed! 
Usage Exception: Unable to apply patch!

Might need to be re-based on the current state of master. But you really should look into using arc; it makes all of this so much easier. Setting it up takes all of 5 minutes. See https://community.kde.org/Infrastructure/Phabricator#Using_Arcanist

shubham updated this revision to Diff 45470.Nov 14 2018, 6:05 PM
shubham retitled this revision from Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 filesystem(4 GB) to Warn user before copy/move job if the file size exceeds the maximum possible file size in FAT32 filesystem(4 GB).
shubham edited the summary of this revision. (Show Details)

Now patch applies cleanly

shubham marked 3 inline comments as done.Nov 14 2018, 6:11 PM
dfaure requested changes to this revision.Nov 21 2018, 12:04 AM
dfaure added a subscriber: dfaure.
dfaure added inline comments.
src/core/copyjob.cpp
1602

This is going to slow down copying in the normal case. This check should only be done if the size is too big (i.e. inside the if further down).

This revision now requires changes to proceed.Nov 21 2018, 12:04 AM
shubham updated this revision to Diff 45951.EditedNov 21 2018, 1:48 PM

Check the filesystem type only if filesize is more than 4GB, otherwise it slows down copying.

shubham marked an inline comment as done.Nov 21 2018, 1:49 PM
dfaure accepted this revision.Nov 21 2018, 3:45 PM

@bruns @ngraham what's your view on this? Did you tested it out?

elvisangelaccio added inline comments.
src/core/global.h
249

Should be bumped to 5.53

@ngraham @bruns do you have any objections?

Sorry for the extended review time; I've found myself super busy with family stuff during this vacation. :)

Please address Elvis' inline comment and then this will look good to me.

src/core/global.h
249

You still need to change @since 5.52 to @since 5.53

shubham updated this revision to Diff 46174.Nov 25 2018, 6:09 AM

@since 5.53

shubham marked 2 inline comments as done.Nov 25 2018, 6:10 AM
dfaure accepted this revision.Nov 25 2018, 8:04 AM
ngraham accepted this revision.Nov 25 2018, 3:03 PM

@bruns is it good to go?

bruns added inline comments.Nov 25 2018, 10:52 PM
src/core/job_error.cpp
1079

"transeferred"? - not done ...

shubham updated this revision to Diff 46235.Nov 26 2018, 5:44 AM
shubham marked an inline comment as done.

"transferred" done
request @ngraham @dfaure @bruns to give thumbs up again.

ngraham accepted this revision.Nov 26 2018, 5:47 AM

Waiting for green flags.

bruns accepted this revision.Nov 29 2018, 12:38 PM
This revision is now accepted and ready to land.Nov 29 2018, 12:38 PM

Also +1 from me.

However, I would strongly suggest waiting till after Saturday when frameworks is tagged.
Then you'll have a month for translators and for some testing.

I agree. Because this contains new strings and we're so close to the 5.53 release, let's land this for Frameworks 5.54 after tagging.

@shubham I'm afraid this means you'll now need to change all the instances of 5.53 to 5.54! 😛 I know it's taken a long time, but don't worry, it will be landed soon...

shubham updated this revision to Diff 46490.Nov 29 2018, 4:46 PM

Change to KF 5.54
Will land it after saturday. This patch actually missed 2 tagging (5.52 and 5.53)

Awesome, can't wait to have this in!

This can land now!

This revision was automatically updated to reflect the committed changes.