Find Jobs
Hire Freelancers

Build a python module that can stream data and extract it to then store it in a MySQL

£20-250 GBP

Fullført
Lagt ut over 4 år siden

£20-250 GBP

Betalt ved levering
I want to have a python project which can consume a stream of data from a KAFKA topic. For example, when you send this command: bin/[login to view URL] --bootstrap-server localhost:9092 --topic myTestTopic --from-beginning It will send an output like this: (there will be 1000's of these lines.... just taking top 2 for this example) &time=1561768216000&gameCategory=PINPOINT&category=ONE&uniqueId=2518Z-0892A-0030O-16H70&transactionType=CRD&familyId=000-222-115-11119&realTs=1561768319000&sortId=1&msg=SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes.&remoteIpAddress=127.0.0.1&userAgent=HTTP& &uniqueId=872541806296826880&time=1571988786000&gameCategory=NOTIFY&category=TWO&transactionType=CRD&familyId=401-222-115-89387&sortId=1&realTs=1571988989000&msg=This-is+a+reminder.&remoteIpAddress=127.0.0.1&userAgent=HTTPS& I want to consume the following from the output: -realTs -familyId -msg -uniqueId and you can see that each element is seperated by an ampersand ('&'). They are not always in same index/place so I'm not sure if I need a regex? These will be processed and batch inserted. I want this batch to be of size 100 let's say which can be configurable from a properties file. Eventually when I do the query on a local running MySQL, i'd see this: SELECT * FROM testTable; +---------------+-------------------+-----------------------------------------------------------+-------------------------+ | realTs | familyId | msg | uniqueId | +---------------+-------------------+-----------------------------------------------------------+-------------------------+ | 1561768319000 | 000-222-115-11119 | SET-UP+PRAYER+%26+intercession+begins+in+just+30+minutes. | 2518Z-0892A-0030O-16H70 | | 1571988989000 | 401-222-115-89387 | This-is+a+reminder. | 872541806296826880 | +---------------+-------------------+-----------------------------------------------------------+-------------------------+ It will also extract value of "time" element of the last line in a batch, and store this in another MySQL table called STATE. This table will only have 1 column: SELECT * FROM state; +---------------+ | time | +---------------+ | 1571988786000 | +---------------+ Once this is done, I will have another python module in same project that will connect to the MySQL database and extract data from MySQL table. It will take an argument and it will start extracting all rows from a given (realTs) and print them to console. So, in total, 2 modules. I want to keep code clean and keep passwords and other configs in properties file. 1) One which streams from KAFKA topic and extracts and processes it to store in MySQL 2) One which then connects to this local MySQL and starting from a given realTs then print whole row to console.
Prosjekt-ID: 22887347

Om prosjektet

5 forslag
Eksternt prosjekt
Aktiv 4 år siden

Ønsker du å tjene penger?

Fordeler med budgivning på Freelancer

Angi budsjettet og tidsrammen
Få betalt for arbeidet ditt
Skisser forslaget ditt
Det er gratis å registrere seg og by på jobber
Tildelt til:
Brukeravatar
Hello? How are you? I am excited to work with you on this project. I have done a lot of jobs with python like Django admin, Flask, python scrap, pysql, python tkinter GUI etc Here is on of my scrap with python working result. [login to view URL] [login to view URL] This is just [login to view URL] sport site scrap. [login to view URL] I can do python with selenium, Beautifsoup, scrapy etc... So i hope to discuss your project on chat deeply. Thanks. Best regards.
£150 GBP om 3 dager
4,9 (76 omtaler)
6,5
6,5
5 frilansere byr i gjennomsnitt £171 GBP for denne jobben
Brukeravatar
Hello, I'm David, I can provide you perfect work as you looking for . we can discuss more in chat . I will be waiting for your response . I have very great experience about your project , if you want discuss more please send me a text Thank you
£135 GBP om 7 dager
0,0 (0 omtaler)
0,0
0,0
Brukeravatar
I have a lot of experience using python, I have a degree in Computer and electronic engineering from the university of Cape Town and I am currently doing a data science course. I have worked with various frame works on python from data processing to web hosting
£222 GBP om 5 dager
0,0 (0 omtaler)
0,0
0,0

Om klienten

UNITED KINGDOMs flagg
Hammersmith, United Kingdom
5,0
2
Betalingsmetode bekreftet
Medlem siden des. 15, 2019

Klientbekreftelse

Takk! Vi har sendt deg en lenke for at du skal kunne kreve din gratis kreditt.
Noe gikk galt. Vær så snill, prøv på nytt.
Registrerte brukere Publiserte jobber
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Forhåndsvisning innlasting
Tillatelse gitt for geolokalisering.
Påloggingsøkten din er utløpt og du har blitt logget ut. Logg på igjen.