Bigquery to kafka jobber
Requirement: experience in Node.js building applications and experience in migrating the monilithic application to microservices architecture applications. Experience in kafka for queuing. Experience in mongodb for database and angular for frontend. PS: This project is a very good opportunity for the developers who wants to learn development from scratch using modern micro service architecture This is a to support my job opportunity. Consultancies and companies please stay away. Need someone with good experience on NodeJs to help me understand my project, resolve blockers and provide constant feedback to improve myself.
...are seeking a skilled developer to set up and program a framework using Apache Kafka for distributed messaging. The project aims to establish an efficient and scalable messaging system that can handle high-volume message processing and filtering. Project Overview: The developer will be responsible for setting up Apache Kafka on a Windows environment and developing a master script and worker scripts to facilitate distributed messaging. The key requirement is to implement a mechanism for bulk-reviewing messages in the queues every 10 minutes to filter out unnecessary messages and optimize system performance. Responsibilities: Apache Kafka Setup: Install and configure Apache Kafka on a Windows machine. Set up the necessary ...
My project requires an experienced Java SpringBoot developer with a strong understanding of Kafka for asynchronous communication. The ideal candidate should also have: - Proficiency in building and managing REST APIs, focusing on CRUD operations - Ability to integrate with external APIs and service data repository calls - Expertise in crafting and handling SQL queries, preparing for moderate traffic and frequent daily queries - Competence in setting up batch schedulers - Proficiency in sending email notifications This project offers an excellent opportunity to apply and enhance your Java SpringBoot and Kafka skills.
Looking for a skilled developer to assist with configuring Apache Kafka for a distributed messaging system. I have running zookeeper, kafka-server and kafka-connect. the project requires: 1) configure kafka source connector for receiving a Json and store it on a topic. 2) configure a kafka sink connector for storing the json data into a PostgreSQL table 3) guidance for installing the plugins 4) setup the connectors over kafka-api the kafka server will be running on ubunut 24.x without a docker Ideal Skills and Experience: - Familiarity with Apache Kafka platform - Experience in distributed messaging - Strong problem-solving skills - Attention to detail We are eager to work with someone who is able to...
Seeking a skilled and experienced Data Analyst to sift through unspecified data sources. As the data type, desired insights, and current storage system have not been specified, I am looking for someone highly flexible and adept at handling all types of data, formats, and analysis needs. Ideal Skills and Experience: * Proficiency with Excel, database, BigQuery, JSON, and CSV file formats * Experience working with sales data, financial data, and customer data * Ability to identify sales trends, perform customer segmentation, and analyze profitability Please note that the specifics will be provided upon project commencement. We look forward to hearing from data analysts who can adapt to different data types and uncover valuable insights.
...com/in/puneet-bajaj-iitm/ These are my details Name - PUNEET BAJAJ ACHIEVEMENTS JEE Main 2023 : Achieved a 99.29 percentile, reflecting dedication and effort. JEE Advanced 2023 : Among top 11k students out of 1.9 lac test takers NAEST 2022 : Among only 1000 students selected out of 1.15 lakh from 8th to M.Sc. IOQM 2022 : Qualified Part A of toughest Math exam for students & wrote INMO . Harvard Ready Player 50 : Among only 3700 across the world to solve all 7 levels . CERTIFICATIONS: Google Advanced Data Analytics Professional Certificate | Google - View Credential Google Cybersecurity Professional Certificate | Google - View Credential Machine Learning Engineering for Production (MLOps) Specialization | - View Credential Deep Learning Specialization
I'm in search of a seasoned DevOps consultant with strong expertise in Kafka and Akka. The role centers around the design, implementation, and monitoring of our cloud-based infrastructure on AWS. Strong scala programming skills and experience with Scala frameworks such as Akka. Efficient in designing and developing REST API’s. Experience with SQL and NoSQL databases. Proficiency in software design patterns and principles. Experience with version control tools such as Git. Good understanding of Agile methodologies and a collaborative mindset. Excellent problem- solving and analytical skills. Experience in unit testing (eg. JUnit/ Mockito). Experience in cloud- based environments such as AWS or Azure. Experience with Kafka and Elasticsearch. Experience : 5+ years ...
Google big query - for writing sql queries for the complex data
I'm in need of a proficient Node.js developer for a project that involves Kafka. Here's an overview of what I'm looking for: The ideal candidate should have experience with: - Node.js - Kafka - Building, implementing, and debugging Kafka workflows Please get in touch if you can meet these requirements.
... The ideal candidate for this project should be able to: - Perform data analysis: This includes understanding, interpreting, and presenting data to inform decision-making. - Manage data integration: The candidate should be capable of combining data from various sources to create a cohesive dataset. - Develop data visualization: The engineers will need to create meaningful and understandable visual representations of the analyzed data. The necessary skills for this role are: - Proficient in Python: The candidate should be well-versed in Python programming language to work with our system. - Knowledge of Hadoop: Experience with Hadoop is a must for this position to handle the data integration and data storage tasks. - Knowledge of Kafka ...
Project Description: We are looking for a skilled Java developer with experience in Apache Flink (version 1.19) to create a data processing job that reads from parquet files located on a Network File System (NFS) and sends selected data to a Kafka topic. Core Responsibilities: - Develop a Flink job using Java that can read parquet files from NFS. - Implement functionality to allow the selection of specific columns from the parquet files to be processed. - Ensure the selected data is published to a specified Kafka topic efficiently. - Manage dependencies and libraries required to execute the job without compatibility issues.
I'm seeking an expert who can help me integrate Kafka and UiPath for the purpose of automating specific business processes that involve financial transactions. Key Responsibilities: - Develop a seamless integration between Kafka and UiPath, using code stage in C# or .Net Skills and Experience: - Proficiency in Kafka and UiPath - Proven track record in automating - Strong understanding of data streaming and business automation - Ability to provide clear documentation for the implemented solution Please bid if you have relevant experience in this domain and can deliver a robust and efficient solution for our needs.
**Project Scope** Seeking skilled Data Analyst specializing in PUBG Mobile to elevate our team's performance using data insights. Project goals: * Performance Baseline: Deep understanding of our team's key metrics (individual players & team stats). * Competitive Benchmarking: Analyze opponents to uncover strengths, weaknesses, strategic tendencies. * Optimal Rotation Analysis: Decipher winning map rotation patterns from videos. Find areas we can improve. * Data-Driven Recommendations: Provide clear report on insights and suggest changes to team composition, drop spots, in-game decisions. **Project Deliverables** * Structured dataset in Google BigQuery with our team's gameplay stats + relevant opponent data. * SQL queries designed for meani...
...a database startup called and we need to write articles comparing the architecture of our database with other popular analytical databases. We also need some articles related to our customer success and use cases. The objective here is to craft articles that spotlight and explain the differences in the architectural methodologies between our startup database, , and other well-known analytical databases - specifically Cassandra, BigQuery, Snowflake, and Redshift. Key Responsibilities: - Research, understand, and articulate the distinct approaches of and the other specified databases - Translate complex concepts into clear, concise, and reader-friendly articles Ideal Candidate Should Have: - You need to have very deep expertise in databases and dist...
...Idea: Real-time Sentiment Analysis for an Online Store 1. Overview: - Develop a web application for an online store where users can submit reviews for products. - Utilize Apache Kafka for real-time event streaming. - Implement Apache Airflow for workflow orchestration. - Use Apache Spark for data processing and sentiment analysis. - Store processed data in Cassandra DB for recording activities. - Store analyzed data in Amazon S3. 2. Workflow: - User submits a review through the web UI. - The review data is streamed to Kafka topics. - Airflow schedules tasks to ingest data from Kafka, process it using Spark, and store it in Cassandra. - Spark performs sentiment analysis on reviews. categorize it into positive , negative,...
I'm seeking a proficient developer to build a comprehensive demo or starter code integrating Snowflake and Kafka using AWS cloud. The ideal candidate should have deep understanding of concepts such as: - Data ingestion - Data transformation - Data storage and retrieval The code should ensure: - Optimal performance - Scalability - Security The code can be written in Python or Java. In-depth knowledge of these languages, as well as experience with Snowflake and Kafka integration, is a necessity for a successful bid on this project.
Je suis en train de développer un mini projet qui intègre Jupyter et Kafka avec Spark Streaming pour le traitement des données, et utilise Cassandra pour le stockage. Pour la visualisation et la gestion des alertes, j'emploie Grafana. Le projet inclut également une fonctionnalité spécifique pour afficher la température de différentes régions, gérée de manière aléatoire. Je suis à la recherche d'une personne capable de prendre en charge le développement de ce projet. Je suis disposé à rémunérer pour ce service. Si vous avez l'expertise requise et êtes intéressé par cette opportunité, n'hésitez pas &a...
I'm using Kafka 2.6.0 with a sink connector to Postgres on Confluent Cloud. The issue I'm facing is due to extensive modifications to the default configurations for the sink connector. Key Points: - The primary issue I'm encountering is related to the configuration of the sink connector. - The connector is not functioning correctly, and data is not arriving in Postgres as expected. - I have made significant modifications to the default configurations for the sink connector. I need an expert who is proficient in Kafka, Confluent Cloud, and sink connector configuration to help debug and rectify the issue. You should be able to: - Analyze and troubleshoot the sink connector configuration to identify the ...
...C++ programmer to develop a Windows application and backend service that can help in calculating features for real-time streaming of stock tick data. This application needs to be proficient in: - Maintaining the full lifecycle of stocks for specific time periods like pre, trading, and closing per day. - Customizable day feature(calculate in startOfDay) and tick feature (calculate when tick comes in) real-time calculation. - High performance for calculating simultaneously. - An API that can serve all features for a specific stock for the current day for training input - Support for customizable trading logic triggering based on market snapshot features every N seconds. - Monitoring progress status (per day, and days) - support backtesting using mongodb data or kafka ...
Seeking a skilled developer to optimize and enhance the architecture of our existing web scraper application. The application is currently built using NestJS and PostgreDB, and we are looking to scale it up and leverage cloud functionality for improved performance and efficiency. Responsibilities: - Review and analyze the current architecture and codebase of our web scraper application. - Propose and implement architectural improvements to optimize scalability, performance, and maintainability. - Evaluate and recommend suitable cloud technologies and services for our scraping solution. - Migrate the application to a scalable architecture, such as serverless, containerization, or a hybrid approach. - Implement distributed scraping techniques using message queues...
...Technical: Master's degree with 10 – 12 years related experience, of which at least 7 + years in development, 4+ years of product engineering experience Experience in Analytics and Business Intelligence is a plus Strong design and coding experience Expertise with the Go Lang platform or higher Experience with integration, data handling technologies and other open source frameworks such as Spring, Kafka, Elasticsearch, Solr etc Experience in NLP, Data Science and Text analysis is a plus. Good Knowledge for Python and GoLang. Experience in building event driven cloud native micro services architectures Experience in building ELK stack Hands on experience in Mongo, Redis, Cassendra, Neo4j, Postgres is a plus Experience with massive-scale performance and scalab...
...AI restaurant recommendations platform and looking for a developer to take on work as required. I am a full stack engineer and built the platform on my own so this request is more for bringing in a partner to take on some new work. Status updates will be expected everyday and tracked via JIRA. The platform is built with React on the front end with Java on the backend and it is deployed on GCP. There is work across the board of the platform including the writing of test cases for the front end and backend of the platform. Please provide your strategy to implement these in the application as well as how long it would take. This role while not in consideration for full time might be a long term engagement. Please provide links to your repo if available. My ide...
I need someone to create the dag and trigger it I am working on a migration project from hadoop to bigquery....more details will be shared via chat
I'm looking for an expert in GCP & Airflow who understands the intricacies of creating an Airflo...creating an Airflow DAG in Project-A (cloud composer) and triggering it via Project-B's service account. Work encompasses: - Crafting an Airflow DAG implementing Python script to create BigQuery tables. - Making the DAG operative within the confines of Project-A. - Leverage the service account of Project-B to trigger the DAG. - The BigQuery tables need to be created within Project-C. Ideal skills and experience: - Proficient in Python Scripting. - Experience in creating and managing workflows using Airflow. - Extensive knowledge of Google Cloud Platform. - Familiarity with BigQuery is a must. - Proven ability to script and ma...
...Services Domain Consulting Staffing augmentation-Short term and Long term Production Support using Service Now, BMC Remedy and other leading tools(24 *7) DepartmentInformation Technology Grade:N/A Experience: 10 to 12 years Job Type: Full Time number of Positions: 1 Location : Pune Job Description About job:- Lead Engineer to oversee and guide the delivery of Safe AGILE programs. We are seeking a candidate with extensive and up-to-date hands-on experience in full-stack technical development encompassing Java, Cloud, Web, UI, and mobile technologies, as well as exposure to Domain-Driven Design, Event-Based Architecture, API, and microservices architecture. The ideal candidate should possess a strong enthusiasm for emerging technologies and demonstr...
Looking for crash course on "Apache Kafka" Tutor
As a client, I'm seeking a Python developer who that can help me trasforming data. The primary source of the data will be company-related information. Ideal Skills and Experience: - Proficiency in Python - Extensive experience in data analysis, par...primary source of the data will be company-related information. Ideal Skills and Experience: - Proficiency in Python - Extensive experience in data analysis, particularly with telecomdata - Familiarity with APIs for data collection - Knowledge of telecom - Ability to understand Python application and python scripts -Know about Airflow , Spark, Kafka, Transaction Mysql Key Requirements: - Develop a Python program that can efficiently trasform non useful to useful data. - Strong understanding of telecom data -Spar...
...of a Full-Stack developer well-versed in Apache Kafka for configuration, message serialization, deserialization, event streaming, and processing. Additionally, you will be responsible for developing Spring Boot microservices that encompass user authentication, authorization, database integration, data modeling, and REST API development. Key responsibilities include: - Kafka setup and configuration - Kafka message serialization and deserialization - Kafka event streaming and processing - Designing and implementing user authentication and authorization within Spring Boot - Integrating databases and modeling data effectively - Developing REST APIs with Spring Boot The ideal candidate for this project should have extensive experience with Kafka and Spri...
I need a Node.js developer with expertise in transitioning a fully operating monolithic application to a microservices architecture using Kafka, Redis Server, and Openshift. A successful migration will preserve current functionalities including: - User profile management - Transactions processing - Content management The deadline is paramount, thus a professional with great time management skills is ideal. Prior experience in similar projects is highly desirable. Your expertise will ensure a seamless transition with zero interference to the current users.
Develop a real-time data integration and processing system that synchronizes data from a MongoDB database to Google BigQuery using Google Cloud Dataflow, ensuring that any new collections added to MongoDB are automatically transmitted to BigQuery without manual intervention.
I am seeking a technically adept developer skilled in Java, Typescript, and Kafka for the purpose of integrating multiple technologies. This project is primarily focused on setting up a microservice architecture to efficiently process transactional data. Your responsibilities will include: - Developing robust Kafka producers and consumers to enable efficient data flow - Utilising your Java and Typescript expertise to build functional microservices Strong experience in Java, Typescript, Kafka, and microservice development is vital. An understanding of transactional data processing would be beneficial. Keen attention to detail, problem-solving skills, and a drive for tech integration efficiency are qualities sought for this project.
I'm looking for an expert to assist me in analyzing sensor/IoT data using BigQuery on Google Cloud Platform. Key Responsibilities: - You will be responsible for conducting data analysis on our sensor and IoT data. - Experience in Google Cloud Platform and BigQuery is a necessity for this project. - I am looking for someone with a strong background in data analysis, specifically with sensor and IoT data. Ideal Candidate: - Proficient in Google Cloud Platform, specifically BigQuery - Previous experience with sensor and IoT data - Strong data analysis skills - Ability to communicate results effectively We're looking for someone who can not only analyze our data effectively but also communicate the results in a way that is easily understoo...
...legacy infrastructure and frameworks such as Scalding to more modern infrastructure we support such as Spark Scala. This role will be responsible for: Analyzing existing data pipelines to understand their architecture, dependencies, and functionality. Working closely with data engineers to develop a migration strategy for converting pipelines from the current framework to the target framework. Designing, building, and testing new data pipelines in the target framework, ensuring they meet or exceed existing performance and reliability standards. Debugging and troubleshooting issues that arise during the migration process, working collaboratively with cross-functional teams to resolve them quickly. Communicating progress, challenges, and timelines to...
Hi, Please apply only individual. Agency can apply but budget should not be more than mentioned. Role : GCP Engineer (OTP) Exp : 7 + yrs SHIFT: IST Cloud Storage Buckets, BigQuery (SQL, Data Transformations and movement) Airflow (python, DAGs), DBT IAM Policies PyCharm Databricks (pySpark), Azure DevOps Clear and confident communication
...legacy infrastructure and frameworks such as Scalding to more modern infrastructure we support such as Spark Scala. This role will be responsible for: Analyzing existing data pipelines to understand their architecture, dependencies, and functionality. Working closely with data engineers to develop a migration strategy for converting pipelines from the current framework to the target framework. Designing, building, and testing new data pipelines in the target framework, ensuring they meet or exceed existing performance and reliability standards. Debugging and troubleshooting issues that arise during the migration process, working collaboratively with cross-functional teams to resolve them quickly. Communicating progress, challenges, and timelines to...
Vennligst Registrer deg eller Logg inn for å se detaljer.
We are seeking a highly skilled Cloud-Native Application Architect for a two-week engagement to design and define the architecture for a new visualization and analytics platform. Responsibilities: • Collaborate with stakeholders to gather and understand visualization and analytics requirements for the platform. • Design a comprehensive cloud-native architecture for the visualization and analytics platform, encompassing data ingestion, processing, storage, visualization, and analytics components. • Leverage your expertise to recommend open-source technologies, off-the-shelf software tools, or a combination of both to implement the chosen architecture. • Translate the overall architecture into a specific AWS cloud implementation plan. o Select ...
...developer for a LONG TERM cooperation for a developer to participate in the product development PLEASE EXPLAIN IN YOUR ANSWER WHY YOU WOULD LIKE THIS OPPORTUNITY AND WHY YOU ARE A GOOD CANDIDATE FOR IT We would like to have excellent skills in - Angular (version 11+) - Java springboot - Ionic Mobile Apps - MySQL - Kafka Other good to have skillls - Understanding of e-commerce catalog & order data model - Payment provider integration - Bluetooth & POS printer connectivity - Java FX for desktop application - nice to have No agencies, please! Ideally, we would like a developer that would be able to dedicate 20 - 40 h per week, so it is ok to have another job too but 8h/day can be considered Must have - Excellent English / commun...
...looking for a skilled freelancer or agency to elevate our current Google Sheets-based systems into a robust SQL or BigQuery database framework. The objective is to create a user-friendly web dashboard that seamlessly interacts with the database, ensuring smooth data updates and retrieval. .Collaborate with cross-functional teams to understand current Google Sheets systems and user requirements. .Design and develop a web dashboard for user interaction with the SQL or BigQuery database. .Implement user authentication and authorization protocols for secure access to the system. .Create a system where users can create unique usernames and perform tasks associated with their roles. .Establish data synchronization between the web dashboard and the ...
I'm seeking a skilled freelancer to convert a geojson file to New Line json format. The purpose of this conversion is specifically for use within Looker Studio. The file will be stored in BiGQuery. The file has to be accepted in BigQuery and be useable in Looker Studio
I need a proficient Google Cloud engineer to assist in the verification of my Google Cloud project application with external Google accounts. Key Responsibilities: - Rigorous understanding and use of Compute Engine, Cloud Storage and BigQuery. - Verification of my Google Cloud project specifically on the OAuth Consent Screen. - Integration of the Google Cloud project with external Google accounts. - Signature of non-disclosure agreement. - Complete project within 16-30 hours. Ideal candidate must be Google Cloud certified, with extensive experience in application integration and account setup. Proven problem-solving skills would be beneficial.
...experienced developer who specializes in Kafka and Genesys system integration. The following is the tasks and main aim for this project: i) Deploy (Apache) Kafka on existing YTLC CES server and ensure it is compatible with the required Co-Browse version ii) Integrate Genesys Co browse to Kafka for it to send the Co-Browse Reporting Event to Kafka iii) Integrate Infomart to Kafka for it to extract data from the Kafka data stream and transforms it into the COBROWSE_FACTtable iv) To ensure / enable RAA to summarize and organizes the Info Martdata in ways that enable GI2 to extract meaning. v) To support required Custom Report template based on available reporting data field and ensure ...
...experienced developer who specializes in Kafka and Genesys system integration. The following is the tasks and main aim for this project: i) Deploy (Apache) Kafka on existing YTLC CES server and ensure it is compatible with the required Co-Browse version ii) Integrate Genesys Co browse to Kafka for it to send the Co-Browse Reporting Event to Kafka iii) Integrate Infomart to Kafka for it to extract data from the Kafka data stream and transforms it into the COBROWSE_FACTtable iv) To ensure / enable RAA to summarize and organizes the Info Martdata in ways that enable GI2 to extract meaning. v) To support required Custom Report template based on available reporting data field and ensure ...
I am looking for a skilled developer experienced in Spring Boot, Kubernetes, Dockers, Kafka, and AWS deployment to build a microservices app. This project requires: - User Authentication: The app should be able to handle user signup, login, password recovery, and security checks. - Data Processing: Facilitating data collection, organization, processing, and efficient management. - Third Party Service Integration: Seamless interoperability with other external applications or systems for enhanced functionality. The app is geared towards the general public and business-to-consumer (B2C). It is therefore essential that the app is user-friendly and aligns to the best industry practices for B2C applications. The ideal freelancer should have a proven trac...
...a highly skilled Kafka expert with deep expertise on real production environments. We want to have a high available solution which is also fail tolerant to destination processing systems Strong expertise on Kafka and how to read data from one system, parse this data reliably with options to debug later and finally post data to the destination system What is not required: - HA solutions of kafka - high load - failsave Budget? will not be disclosed, place your best bid to get considered What is next? We will share you a NDA and afterwards a paid test task. Payment? - you estimate in a WBS (optimistic, expected, pessimistic, where optimistic < expected < pessimistic) after getting the task - we disc...
For this task, I require a skilled developer with solid background in Docker and Kafka. The task at hand is to work on an existing Docker Compose script for a Kafka application centered around authorization. Key Requirements: - Incorporating SASL/SCRAM authentication mechanism. - Focusing on the SCRAM-SHA-256 version for Kafka authentication. A deep understanding of the listed technologies and profound experience in adding security features to Kafka applications will be a significant advantage. Secure, reliable streaming is the goal. Therefore, the freelancer needs to be knowledgeable about how SASL/SCRAM authentication enhances Kafka application's security. Only bid if you are highly confident you can deliver my exact requir...
...talented freelance data engineer to join our team on a project basis. The ideal candidate will have a strong background in data engineering, with expertise in designing, implementing, and maintaining data pipelines and infrastructure. You will work closely with our data scientists and analysts to ensure the smooth flow of data from various sources to our data warehouse, and to support the development of analytics and machine learning solutions. This is a remote position with flexible hours. *Responsibilities:* - Design, build, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of data from diverse sources. - Collaborate with data scientists and analysts to understand data requirements and implement s...
I need a Airflow expert to help redesign my data flow and put in place a solution that is scalable and can account for future needs. Our basic flow is making API calls to third-party solutions and getting data for various customers. This requires using the right API keys for each customer. The data will then be put in a Kafka queue for further processing. I am looking for someone with experience to help think through and implement solutions.
...am looking for a skilled developer proficient in Apache Camel, Spring Boot, Kafka, and Ksqldb. The goal is to integrate Salesforce, ERP, and SAP using a cloud to cloud rest api approach. The successful applicant will have extensive experience leveraging these technologies to implement both staging and production deployment. Key Responsibilities: - Build a middleware solution for cloud to cloud rest api integration. - Ensure this solution adheres to best practice for handling transactions. Specifics: - An existing REST API for the Salesforce integration is present. - The Salesforce integration needs to support order management functionality. Ideal Skills: - In-depth knowledge of Apache Camel, Spring Boot, Kafka, Ksqldb. - P...
