Bash script to read text values in files recursively and prepare a new file ready for MySQL

We have a series of text files which contain lists of log file information in this format ->



There is no delimitation around the fields other than the full colon ":" separating the 2 values and obviously a new line at the end of the 2nd value. There are however some errors in these files, some of the lines don't follow the correct format above and we need these removing otherwise we wont be able to use LOAD DATA into MySQL when we come to import them.

These files are stored in multiple nested directories, some of which are 5 or 6 levels deep and each directory can contain between 50 and 200 text files.

We are looking for someone to write a simple BASH script to run on ubuntu 16.04 in the root directory of all these files. The script will traverse its way down each file and directory and cleanse the data. The script will need to open each file, read it one line at a time and check that each line conforms to the above format. If it is in the correct format, add it to a new "cleansed data" text file in the same (value1:value2) format in the current directory being checked. When its finished reading all the files in that current directory, it will save the cleansed data text file and move onto the next dorectory.

We have 98 directories of this data and hundreds of files within them. The output of the script will be 98 "cleansed" text files, one in each directory containing all the correctly formatted lines from all the files in that directory.

We'll then use each new cleansed data file with the LOAD DATA facility to upload the new value1:value2 checked pairs.

Ferdigheter: Linux, MySQL, Shell Script

Se mer: shell script to read a file line by line and write to another file, how to read a file line by line in unix using for loop, shell script to read a file line by line in unix, bash while read, shell script to read data from text file, bash read file line by line, bash read first line of file, bash script read file into variable, script unzip files recursively, script read files, script read text pdf, script read text file, bash script read logs, bash script read columns spreadsheet, script read text sentence, vba script read text file, script read text files, vbs script read text file export html file, lotus script read text files, script find text xml files

Om arbeidsgiveren:
( 1 anmeldelse ) Lincoln, United Kingdom

Prosjekt-ID: #18159808

Tildelt til:


HI, I have ten years commercial Linux experience as a systems administrator. I can write a script to filter your files. Hope to speak with you soon.

£23 GBP på 1 dag
(14 omtaler)

11 frilansere byr i snitt £57 for denne jobben


Bash is one of my best skills. I use it every day. All I would need is a few sample files, and some nodes about data types in the file.

£20 GBP på 1 dag
(245 Omtaler)

I have worked as server admin for big names like [login to view URL] and servercentre.net. Over 8yrs of working with cpanel, directadmin and plesk servers. Can help you with this. Let me know.

£50 GBP på 1 dag
(124 Omtaler)

Hello I can do it immediately in python Thanks ...............................................................

£111 GBP på 1 dag
(2 Omtaler)

Hello Sir, I am agile safe 4.0 certified I have 7 year of experience in Software development including technologies java, j2ee, spring , hibernate, ,Rest apis, php unix shell scripts, apache tomcat, jboss, server co Mer

£10 GBP på 1 dag
(3 Omtaler)

Hi, there. I am happy to apply in your project. Your project is interesting to me. An experienced Bash/Shell script expert here. If you want to review me, you can go through my feedbacks and git repos. https://git Mer

£222 GBP på 1 dag
(7 Omtaler)

Hi, I can write a python script for this that will full fill the purpose. As you will require data cleanse, so I feel no perfect script can do. There will always scenarios that are not present in current data but can b Mer

£33 GBP på 2 dager
(3 Omtaler)

Hello, I'm interested in the project. Right now I have a small web development company with 2-3 web developers. We are currently working on the back-end of our website and when we fully launch our front-end will be com Mer

£18 GBP på 1 dag
(0 Omtaler)

Hi, I wrote many such nifty content aware scripts. This is doable with shellscript, ruby, perl and almost any language. I will choose the one that will get the job done best. I hope you decide for me and be amazed!

£55 GBP på 1 dag
(0 Omtaler)

6 Years of Experience, Expert in all kind of applications, Laravel 5.1, 5.5, Spring Mysql, SQL Server.

£23 GBP på 2 dager
(0 Omtaler)

As your description of what the script should do sounds quite complete, as I understand it, the script should: - take as a parameter the root directory of where the log files are - recursively traverse the entire tree Mer

£61 GBP på 2 dager
(0 Omtaler)