Long running process in Linux using PHP

Background

To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages.
A bit of googling in stack overflow found us this http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script, but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation.

Difficulties in long running PHP scripts

I encountered some problems when trying to make PHP do long running tasks :
  1. PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks.
  2. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past) or splitting the data to process into several batches. And if you are going to loop the batches in one PHP run, make sure after each batch there are no dangling reference to the objects processed. 
  3. Browser disconnects in Apache-PHP environment would terminate the PHP script. During my explorations I found that :
    1. Some firewall usually disconnects a HTTP connection after 60 seconds.
    2. Firefox have a long timeout (300 seconds or something, ref here
    3. Chrome have timeout similar to Firefox (about 300, ref here), and longer for AJAX (stackoverflow ref doesnt timeout after 15 hours)
  4. Difficulties in running pentaho transformations, because the PHP module would run as www-data, and will be unable to access the kettle repository stored in another user's home directory.

Workarounds

I have experiences using these workarounds to force PHP to be able to do long running web pages :
  • Workaround 1 : use set_time_limit(0); and ignore_user_abort(true); to ensure script keeps running even after client disconnects.  Unfortunately the user will no longer see the result of our script.
  • Workaround 2 : use HTTPS so the firewall will unable to do layer 7 processing and doesn't dare disconnect the connection. If the user closed the browser then the script would still terminate, except when you also do workaround 1.
I haven't tried detaching a child process yet like , but my other solutions involve separate process for background processing with similar benefits.

Solution A - Polling task tables using cron

It is better to separate the user interface part (PHP web script) with the background processing part. My first solution is to create cron task that are run every 3 minutes, which runs a PHP CLI script which checks a background task table for tasks with state 'SUBMITTED'. Upon processing the task, the script should update the state to 'PROCESSING'. 
So the user interface/ front end only checks the background task table, and when the user orders to, inserts a task there with the specification required by the task, setting the state to 'SUBMITTED'.
When cron gets to run the PHP CLI script, it would check for tasks, and if there any, change the first task state to PROCESSING and begin processing. When processing complete, the PHP CLI script would change the state to COMPLETED.
Complications happen, so we will need to do risk management by :
  1. logging phases of the process in some database table, including warnings that might be issued during processing.
  2. recording error rows if there is any in another database table, so the user could view problematic rows
Currently this solution works, but recently I came across another solution that might be a better fit for running a Linux process.

Solution B - Using inotifywait and control files

In this solution, I created a control file which contains only one line of CSV. I prepared a PHP CLI script which parses the CSV and executes a long running process, and also a PHP Web page which would write to the control file. Inotifywait from inotify-tools will listen on file system notifications from Linux kernel that are related to changes on the control file.
The scenario is like this :
  1. User opens PHP web page, and choose parameters for the background task, clicked on Submit
  2. PHP web page receive the submitted parameters, and write them into the control file, including job id. The user received a page that states 'task submitted'.
  3. A shell script that running inotifywait, will wait for notifications on the control file, specifically for the close_write event
  4. After close_write event received, the shell script will continue, and run PHP CLI script to do the background processing
  5. PHP CLI script reads the control file for parameters and job id
  6. PHP CLI script executes linux process, redirecting the output to a file identified by job id in a specific directory
  7. The web page that states 'Task Submitted' could periodically poll the output file with the job id, and shows the output to the end user (OK, this one I need to actually try later)
  8. PHP CLI returns, the shell script performs an endless loop by going to (3)

Conclusion

By using Linux file system notifications, we could trigger task execution with parameter specified from a PHP web page. The task could be run as another Linux user, provider the user running the shell script. Data sanitization are done by php, so no strange commands could be passed to the background task. 

These solutions are written entirely in open source solutions. I saw that Azure have WebJobs which might fulfill similar requirements that I have, only it is in Azure platform which I never used.

Comments

Unknown said…
This comment has been removed by the author.
Tuhin said…
This comment has been removed by the author.
tejaswini said…
Through this post, I know that your good knowledge in playing with all the pieces was very helpful. I notify that this is the first place where I find issues I've been searching for. You have a clever yet attractive way of writing.big data analytics malaysia
data science course malaysia
data analytics courses
360DigiTMG
Anonymous said…
I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
360DigiTMG data analytics course malaysia
360DigiTMG best data science certification course
360DigiTMG tableau classroom training
360DigiTMG
Anonymous said…
Wonderful post, i loved reading it.
Share more
Gaudeo
Kitsunemusicacademy
Corejoomla
This comment has been removed by the author.
New site is solid. A debt of gratitude is in order for the colossal exertion.
Data Science Course in Bangalore
This was incredibly an exquisite implementation of your ideas
Data Science Training in Bangalore

360DigiTMG said…
Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
data science course
data science course in hyderabad
data science training
360DigiTMG said…
Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
data science institute un hyderabad
data science training in hyderabad
Maneesha said…
Through this post, I know that your good knowledge in playing with all the pieces was very helpful. I notify that this is the first place where I find issues I've been searching for. You have a clever yet attractive way of writing.
data science
360digitmgdelhi said…
Through this post, I realize that your great information in playing with all the pieces was exceptionally useful. I advise this is the primary spot where I discover issues I've been scanning for. You have a smart yet alluring method of composing.
data science courses in noida
EXCELR said…
This is a really explainable very well and i got more information from your site.Very much useful for me to understand many concepts and helped me a lot.Best data science courses in hyerabad
Navyasri said…

hello sir,
thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh
360digitmgdelhi said…
This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..
data scientist course noida
Unknown said…
I think I have never watched such online diaries ever that has absolute things with all nuances which I need. So thoughtfully update this ever for us.
data analytics training in yelahanka
Thakur98 said…
Kim Ravida is a lifestyle and business coach who helps women in business take powerful money actions and make solid, productiveIamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder IamLinkfeeder
Maneesha said…
You need to be a part of a contest for one of the best websites online. I’m going to recommend this website!
data scientist training and placement in hyderabad
Freenom said…
Annabelle loves to write and has been doing so for many years.iamlinkfeeder1 iamlinkfeeder1 iamlinkfeeder1
ghostus said…
gledajte sve najnovije turske srbije na mreži i sva najnovija ažuriranja serija samo na serijama online

https://serijeonlines.net
Annabelle loves to write and has been doing so for many years.Backlink Indexer My GPL Store Teckum-All about Knowledge
Maneesha said…
This is a fabulous post I seen because of offer it. It is really what I expected to see trust in future you will continue in sharing such a mind boggling post
data scientist training and placement
Oxygen Academy said…
Such a very useful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article.Cyber Security in Visakhapatnam. Cyber Security near me
I would like to thank you for the efforts you have made in writing this article. I am hoping for the same best work from you in the future as well..
data scientist training and placement in hyderabad

data science said…


Great to become visiting your weblog once more, it has been a very long time for me. Pleasantly this article i've been sat tight for such a long time. I will require this post to add up to my task in the school, and it has identical subject along with your review. Much appreciated, great offer. data science course in nagpur
Clubf7 said…
Wow! Such an amazing and helpful post this is. I really really love it. It's so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also.
Best Gym in Visakhapatnam
ghostus said…
the content on your blog is very informational and all blogs are well-writeen thanks for sharing them


apk world
latest version
updated version
No Root
extractor apk
apkstick
Varun Lekho said…
The registration fee for Emirates ID card is AED 100 for UAE and GCC nationals for every 5 years for all age groups plus AED 70 for service fees
emiratesids.blogspot.com
emiratesidcards.blogspot.com
How to Make Yahoo My Homepage on Chrome?

If you want to know about how to make Yahoo my homepage on Chrome then open Google Chrome and tap on settings option. Toggle the bar for the option show home button under the appearance section. After that, click on enter custom web address option under show home button option. Choose the address that you want to open while clicking on home button. Here, enter Yahoo page that you want to set as homepage. At the bottom, you will see the option for opening a specific page or set of pages. Tap on add after entering the address that you want to see once Chrome launches. Lastly, you can smoothly add multiple Yahoo pages and it will open in separate tabs.

How to Turn Off Outlook Notifications on Phone?

Check out the steps mentioned in the guide and know how to turn off Outlook notifications on phone. If you are an Android Gmail user, then start by opening the Gmail application and tapping on the local left menu button. After that, scroll down to the bottom and choose Settings. Lastly, click to account and then uncheck the notifications. These are the steps that users need to follow to smoothly turn off the Outlook notifications on their mobile phone. It is advisable to follow the steps properly to perform the function properly.

How to Deal with Microsoft Outlook Search Not Working?

Sometimes third-party add-ins built to boost the user’s productivity and because of this, users encounter Microsoft Outlook search not working issue. If you want to know about the steps to disable add-ins then start by launching Outlook and choose your File tab. Here, click on options and then click on add-ins from the list. A list of the add-ins includes the active, inactive, and disabled add-ins will appear on the screen. Choose COM add-ins from the manage list and tap to go. Here, untick the add-ins that you want to disable and choose to OK. Lastly, you need to see the effect of disabling the add-ins and then you need to restart the Outlook application.
ghostus said…
Best alternatives sites to Watchwrestling.in - Check our similar list based on world rank and monthly visits only on Xranks

watch wrestling
Watch Pro Wrestling
Ely Girl said…
The Ehsaas Emergency Cash Program utilized the phone number 8171 as a helpline for citizens to apply for financial assistance. By sending an SMS to this number, eligible families could apply for the cash grant. The program aimed to provide support to those who were facing economic challenges and required immediate assistance.

Ehsaas Rashan Program 2023 Online
Ehsaas Program BISP 8171

Popular posts from this blog

Reverse Engineering Reptile Kernel module to Extract Authentication code

SAP System Copy Lessons Learned