Automating tasks using Python for sending files through SFTP

Scheduled background tasks are a staple of IT world. For example, some things (programs) should be done in a certain time in each day,  or some procedure should be run a few times in an hour. In this we will discuss how to send a file to another server using SFTP.

Crontab

Most of the time, background tasks are triggered using UNIX-like crontab, there are some alternatives as well but lets say it is out of current post's scope. The crontab entry usually calls a shell script that in its own triggers other program, such calling a  web site using wget or curl, or invoking application command (php yiic commandname). 
The existing crontab entry for sending the file to another server uses a shell script to prepare the files, create control files, and sending them across the network using sshpass with input redirection.

The Challenge

The problem we're facing is that sometimes the remote server doesn't respond normally. The cause is still unknown, and after getting incomplete files in the remote end usually we ends up resending them (which means there is some business consequences of late processing). There is a probability that network issue is the culprit. In the application side in other hand, the shell script doesn't do retry until about 2 hours later.

Transforming to Python

The idea is that the retry should be done as soon after error message is being detected. Because the author has limited capability in enhancing shell script logic, the author choose to create a python script to do exception handling and retry.
Before coding the solution in python, a due diligence is in order to find out which python library could be used to do the task at hand (i.e. sending a file using SFTP protocol). At first the author tried to use SCPClient  in scp package (https://github.com/jbardin/scp.py / https://pypi.org/project/scp/), but finds out that the remote server blocks SCP functionality and there are no function are provided to check existence a file in the remote or gather remote file size. Then finally the author check the Paramiko sFTP capability ( https://docs.paramiko.org/en/stable/api/sftp.html ) and decided to use it. To install the paramiko sftp use this command : 

The new automated task using Python

The new python code does the equivalent process of sending the file using SFTP, and do it 5 times in a loop, with the code detecting if the file already exists in the remote server with the equivalent size, and breaking the loop if that is the case. Exception handling wrapped the process, ensuring any error will result the python code still in command (and not exiting the python script). Please refer to this Gist URL : https://gist.github.com/yudhiwidyatama/f6cf9a00b6d2edd4a040ecc95098991a

Aftermath

The resulting information log lines from the python script shows that sometimes the remote server not responding, and after about 3x tries it would reconnect and proceed the process. The 5x retry somehow still suffice, and more thorough investigation will be needed because the failure rate is quite high.

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more