Solve MediaWiki TranslatablePageMoveJob issue
Closed, ResolvedPublic

Description

Moving userbase.k.o/Plasma/Create_a_Look_and_Feel_Package to userbase.k.o/Plasma/Create_a_Global_Theme_Package cause an issue jobrunner.

The error message is:

wikiadmin@edulis:/srv/www/mediawiki/mediawiki-new$ WIKI=userbase.kde.org php   maintenance/runJobs.php
2019-10-26 14:09:05 TranslatablePageMoveJob Plasma/Create_a_Global_Package source=Plasma/Create a Look and Feel Package target=Plasma/Create a Global Package moves=array(137) summary=Part of translatable page "Plasma/Create a Look and Feel Package" performer=Ognarb requestId=db5a35420e8c95359706b6af (id=61685,timestamp=20191026140831) STARTING
PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to allocate 262144 bytes) in /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php on line 1255

Fatal error: Allowed memory size of 157286400 bytes exhausted (tried to allocate 262144 bytes) in /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php on line 1255
PHP Fatal error:  Allowed memory size of 157286400 bytes exhausted (tried to allocate 262144 bytes) in Unknown on line 0

Fatal error: Allowed memory size of 157286400 bytes exhausted (tried to allocate 262144 bytes) in Unknown on line 0

Increasing the memory limit by setting ini_set("memory_limit","512M") in maintainance/runJobs.php doesn't solve the issue and I get the exact same error message (with the same ammount of bytes).

So I think the ini_set function is not respected. Any idea that could cause this function to not be respected?

I also tried to specify the memory available in the cli arguments but I get a segmentation fault in this case:

wikiadmin@edulis:/srv/www/mediawiki/mediawiki-new$ WIKI=userbase.kde.org php -d memory_limit=512M  maintenance/runJobs.php --memory-limit 512M
2019-10-26 14:18:28 TranslatablePageMoveJob Plasma/Create_a_Global_Package source=Plasma/Create a Look and Feel Package target=Plasma/Create a Global Package moves=array(137) summary=Part of translatable page "Plasma/Create a Look and Feel Package" performer=Ognarb requestId=dce23d01f159dc467b950aed (id=61692,timestamp=20191026141827) STARTING
Segmentation fault
ognarb created this task.Oct 26 2019, 2:20 PM
Restricted Application added a subscriber: sysadmin. · View Herald TranscriptOct 26 2019, 2:20 PM
bcooksley added a subscriber: bcooksley.

I've had a play with this, and it seems likely that the TranslatePageMoveJob has specified that it should have a memory limit of 150MB, hence why specifying --memory-limit 512M was required.

As for the segfault, it appears to be happening somewhere within PHP's serialization functions, however the overall stack trace is extremely long (on the order of several thousand lines).
Which makes me think it's an infinite recursion issue.

I've installed XDebug and after setting a recursion limit it yields the following trace:

#0 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(482): Wikimedia\Rdbms\Database::getClass(string, NULL)
#1 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1013): Wikimedia\Rdbms\Database::attributesFromType(string, NULL)
#2 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/loadbalancer/LoadBalancer.php(697): Wikimedia\Rdbms\LoadBalancer->getServerAttributes(integer)
#3 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/loadbalancer/LoadBalancer.php(831): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, boolean, integer)
#4 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(783): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string, integer)
#5 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(764): JobQueueDB->getDB(integer)
#6 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(728): JobQueueDB->getMasterDB()
#7 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(224): JobQueueDB->insertFields(JobSpecification)
#8 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(200): JobQueueDB->doBatchPushInternal(Wikimedia\Rdbms\DBConnRef, array, integer, string)
#9 [internal function]: JobQueueDB->{closure}()
#10 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3267): call_user_func(Closure)
#11 [internal function]: Wikimedia\Rdbms\Database->onTransactionPreCommitOrIdle(Closure, string)
#12 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(49): call_user_func_array(array, array)
#13 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(491): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#14 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(202): Wikimedia\Rdbms\DBConnRef->onTransactionPreCommitOrIdle(Closure, string)
#15 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(336): JobQueueDB->doBatchPush(array, integer)
#16 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(306): JobQueue->batchPush(array, integer)
#17 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueGroup.php(160): JobQueue->push(array)
#18 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(317): JobQueueGroup->push(array)
#19 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(299): DeferredUpdates::enqueueUpdates(array)
#20 /srv/www/mediawiki/mediawiki-new/maintenance/Maintenance.php(646): DeferredUpdates::tryOpportunisticExecute(string)
#21 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3461): Maintenance::{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#22 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3795): Wikimedia\Rdbms\Database->runTransactionListenerCallbacks(integer)
#23 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3582): Wikimedia\Rdbms\Database->commit(string, string)
#24 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3268): Wikimedia\Rdbms\Database->endAtomic(string)
#25 [internal function]: Wikimedia\Rdbms\Database->onTransactionPreCommitOrIdle(Closure, string)
#26 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(49): call_user_func_array(array, array)
#27 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(491): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#28 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(202): Wikimedia\Rdbms\DBConnRef->onTransactionPreCommitOrIdle(Closure, string)
#29 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(336): JobQueueDB->doBatchPush(array, integer)
#30 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(306): JobQueue->batchPush(array, integer)
#31 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueGroup.php(160): JobQueue->push(array)
#32 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(317): JobQueueGroup->push(array)
#33 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(299): DeferredUpdates::enqueueUpdates(array)
#34 /srv/www/mediawiki/mediawiki-new/maintenance/Maintenance.php(646): DeferredUpdates::tryOpportunisticExecute(string)
#35 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3461): Maintenance::{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#36 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3795): Wikimedia\Rdbms\Database->runTransactionListenerCallbacks(integer)
#37 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3582): Wikimedia\Rdbms\Database->commit(string, string)
#38 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3268): Wikimedia\Rdbms\Database->endAtomic(string)
#39 [internal function]: Wikimedia\Rdbms\Database->onTransactionPreCommitOrIdle(Closure, string)
#40 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(49): call_user_func_array(array, array)
#41 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/DBConnRef.php(491): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#42 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueDB.php(202): Wikimedia\Rdbms\DBConnRef->onTransactionPreCommitOrIdle(Closure, string)
#43 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(336): JobQueueDB->doBatchPush(array, integer)
#44 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(306): JobQueue->batchPush(array, integer)
#45 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueGroup.php(160): JobQueue->push(array)
#46 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(317): JobQueueGroup->push(array)
#47 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(299): DeferredUpdates::enqueueUpdates(array)
#48 /srv/www/mediawiki/mediawiki-new/maintenance/Maintenance.php(646): DeferredUpdates::tryOpportunisticExecute(string)
#49 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3461): Maintenance::{closure}(integer, Wikimedia\Rdbms\DatabaseMysqli)
#50 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3795): Wikimedia\Rdbms\Database->runTransactionListenerCallbacks(integer)
#51 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3582): Wikimedia\Rdbms\Database->commit(string, string)
#52 /srv/www/mediawiki/mediawiki-new/includes/libs/rdbms/database/Database.php(3268): Wikimedia\Rdbms\Database->endAtomic(string)

The loop in question appears to start after this:

#2031 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(336): JobQueueDB->doBatchPush(array, integer)
#2032 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueue.php(306): JobQueue->batchPush(array, integer)
#2033 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobQueueGroup.php(160): JobQueue->push(array)
#2034 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(317): JobQueueGroup->push(array)
#2035 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(299): DeferredUpdates::enqueueUpdates(array)
#2036 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(97): DeferredUpdates::tryOpportunisticExecute(string)
#2037 /srv/www/mediawiki/mediawiki-new/includes/deferred/DeferredUpdates.php(114): DeferredUpdates::addUpdate(MWCallableUpdate, integer)
#2038 /srv/www/mediawiki/mediawiki-new/includes/logging/LogEntry.php(788): DeferredUpdates::addCallableUpdate(Closure, integer, Wikimedia\Rdbms\DatabaseMysqli)
#2039 /srv/www/mediawiki/mediawiki-new/includes/MovePage.php(615): ManualLogEntry->publish(integer)
#2040 /srv/www/mediawiki/mediawiki-new/includes/MovePage.php(271): MovePage->moveToInternal(User, Title, string, boolean, array)
#2041 /srv/www/mediawiki/mediawiki-new/extensions/Translate/tag/TranslatablePageMoveJob.php(97): MovePage->move(User, string, boolean)
#2042 /srv/www/mediawiki/mediawiki-new/extensions/Translate/tag/TranslatablePageMoveJob.php(54): TranslatablePageMoveJob->doMoves()
#2043 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobRunner.php(296): TranslatablePageMoveJob->run()
#2044 /srv/www/mediawiki/mediawiki-new/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(TranslatablePageMoveJob, Wikimedia\Rdbms\LBFactorySimple, BufferingStatsdDataFactory, integer)
#2045 /srv/www/mediawiki/mediawiki-new/maintenance/runJobs.php(89): JobRunner->run(array)
#2046 /srv/www/mediawiki/mediawiki-new/maintenance/doMaintenance.php(94): RunJobs->execute()
#2047 /srv/www/mediawiki/mediawiki-new/maintenance/runJobs.php(122): require_once(string)
#2048 {main}

Do we perhaps need an updated version of the Translate extension?

ognarb added a comment.Nov 4 2019, 9:22 AM

I create a upstream bug report: https://phabricator.wikimedia.org/T237163

And according to the Translate extension maintainer, this looks like a Job Queue issue and not a Translate Extension issue.

ognarb added a comment.Sep 6 2020, 2:24 PM

With the new update, we got a new error message :(

[de6e72a5a77c469204198de1] [no req]   Error from line 78 of /srv/www/mediawiki/mediawiki-1.34.2/extensions/Translate/tag/TranslationsUpdateJob.php: Call to a member function clearCaches() on null
Backtrace:
#0 /srv/www/mediawiki/mediawiki-1.34.2/includes/jobqueue/JobRunner.php(299): TranslationsUpdateJob->run()
#1 /srv/www/mediawiki/mediawiki-1.34.2/includes/jobqueue/JobRunner.php(192): JobRunner->executeJob()
#2 /srv/www/mediawiki/mediawiki-1.34.2/maintenance/runJobs.php(92): JobRunner->run()
#3 /srv/www/mediawiki/mediawiki-1.34.2/maintenance/doMaintenance.php(99): RunJobs->execute()
#4 /srv/www/mediawiki/mediawiki-1.34.2/maintenance/runJobs.php(129): require_once(string)
#5 {main}
ognarb closed this task as Resolved.Sep 6 2020, 2:33 PM
ognarb claimed this task.

Ok I applied a simple patch to the code and now it looks like it's working just fine :) I will try to upstream it