Python Script to Download

I am an expert in delivering custom scripts and willing to discuss further about the project specifications.

Hello, This looks like a very interesting project. I have extensive experience with HTTP libraries under Python (I have used urllib2, tornado, requests, mechanize) and with threaded programming (especially the multiprocessing module). If the number of files is in the millions we must first establish how many simultaneous requests you are allowed to make to that specific domain. If that number is in the tens then thread pools (or better yet process pools because of the Python GIL) will probably work. However if the number is in the hundreds the asynchronous HTTP client provided by tornado is a much more elegant and faster solution. If the number of simultaneous requests is in the thousands a solution based on combining tornado and multiprocessing is also a good idea. Can you give me some details about the files? How large are they usually; are they all PDF files? I'm asking because I want to get an idea about the compression level that can be achieved and the disk space needed for this project. If you can provide a few URLs so I can get a better idea about this project I will run a few tests and let you know my opinions; afterwards if you are happy with the results we can continue the collaboration on this project. Thank you for your time! Best wishes, Ionut

I am expert in desired skills for this project and have done similar tasks already. Please get back to me, so I can show you some of the work I have done. I will not be asking for any upfront only pay me when you are satisfied with the progress.

Hi there! I have done this type of work before. Please message me if you could with more information such as the url file list. Thanks! -Steve

Hello, I can write this script for you today. It will split the list in smaller of 10000 entries and after it will download to a folder1 first 10000 files, zip the folder1, delete folder, start again again for folder2, zip folder2 and so on until the end of list. sqlite might be better if there are millions of entries. Please send me at least just a few links to test and I'll send you back the script for test, just to be sure that I can write this script for you Best Regards, Constantin Plaiasu

Hi, starting from the text file is enough. Implementing parallel downloads with python is straight-forward. I would start multiple workers to get their own set of 10,000 files and start downloading them. This way you will finish as fast as possible, and this solution needs only python standard modules, hence runs on most installations. I think the total work will take roughly two hours, so you should get results quickly. Regards from Berlin, Guido

Hi, I have such a script ready to go. My work would be to test it for the first 50 files and then schedule and check every time where we are. Might be some things to think about; - is this 1 site where the files are download from so we might have issues with a firewall blocking this after xxxx requests? - do you have an ftp server ready to receive the files from the batches? - This might be running for a few days, I need to spent a few hours on this but can give a precise estimate during the first runs. Using python to do this and have a server ready which can handle this! Dirk

Hello, shouldn't be a problem. What will be done with the urls that failed to download? Written to another file/database/whatever to try later? Any authentication necessary? KR, Oliver

I am highly interested in this project, I can complete the project as per your expectation and withing the deadline. I am having vast exposure of the technologies you mentioned. I am open to met your budget and deadline.

Hi, I have experience in this type of work. Kindly give the information so that we can start Thanks in advance Karthi palanisamy

Hi, I am a Perl Expert. I do Web Scraping, I have Scraped Websites like Amazon. The same thing can be achieved in Perl too. It will be a command line tool and faster. Just drop me a message for further discussion. Regards Nithin

I've worked with python in a lot of projects but I never get any project from here. I think I can do this. I've worked with Python in a Linux distribution (Linux Mandriva) and in the City Council in the city I live (was a database migration, from lotus notes to postgres, using python). I hope I can help you. Thanks in advance.

La propuesta todavía no ha sido proveída

A proposal has not yet been provided

Sir , I have already had a python script for automatically downloading urls(for scratch a ebook site, links are listed in file), I can modify it to fit your demand ASAP Thank you:-)

I am a Computer engineer. If accepted, this will be my first job on freelancer. But trust me when I say that I know what I am doing and your project is very simple to complete. It will take me less than an hour to complete your project. The work will be very good that you will surely be compelled to leave positive reviews. Thank You.

I have the script to do the same work. I am the first time to get a work in freelancer.com, so I ask for very very little pay.

This should be a very quick project. I could easily have it done by Friday. I have experience with Python and threading.

A proposal has not yet been provided

Hello, I have around 9+ yrs of experience in perl/python/C/C++/C++11. I worked in reputed companies like IBM, Alcatel-Lucent. etc. I wish to take up this project and deliver it on time. Thanks Nagaraj