Face Recognition for Access Control and Surveillance System

We are developing a system that allows people to authenticate with an access control system via just their face. We also have several area cameras that track sightings of people throughout the building. We cluster similar faces together based on the similarity between their vector representations. Clusters of faces can be thought of as a face profile. These face profiles can be associated with authorized users, allowing new sightings of them outside of the exterior doors to be used for granting entry via our access control system. We have a test site with three exterior doors. The site has a total of 12 cameras that provide an RTSP stream (2560x1920 @ 30 FPS, h264). Currently, all cameras are configured to record when motion is detected. The recording is then uploaded to an AWS S3 bucket. There are two types of cameras: 1. Cameras that need to be processed in real-time (3 cameras on the outside of each exterior door) 2. Cameras that can be processed later when system resources are available (9 area cameras) We have a computer with the following specs on the local network with the cameras: - Intel Core i7 10700KF 3.8GHz Processor - NVIDIA RTX 3070 8GB GDDR6 - 16GB DDR4-3200 RAM - 1TB SSD - Running Ubuntu 20.04 We have a Python script that was built to do the following: We have three cameras that need to be processed in real-time. We spawn a thread for each camera: 1. Ensure camera is connected, if not, connect 2. Get frame from camera 3. Add to "preprocessing" queue 4. Repeat We use the following queues to manage the flow of frames through our script: 1. "preprocessing" queue 2. "face detection" queue 3. "face feature generation" queue 4. "final processing" queue The script performs the following processing steps: 1. CPU-intenstive color conversion (YUV to RGB) - per frame 2. CPU-intenstive motion detection - per frame 3. GPU-intensive face detection - per frame 4. GPU-intensive face feature generation - per face (potentially many faces per frame) 5. CPU-intensive face comparisons - per face (potentially many faces per frame) 6. Network-bound Update AWS - per face (potentially many faces per frame) 7. Unlock door - per face (potentially many faces per frame) Processing Steps 1 and 2 can be combined and thought of as frame preprocessing. Each of the camera threads can dump frames into the "preprocessing" queue. When the "preprocessing" queue has availability, we also add in frames from historical footage. Multiprocessing workers can pull frames from the "preprocessing" queue, run step 1+2, and then add the preprocessed frame to the "face detection" queue. That is of course if the frame should be moved forward. Some frames, like frames without motion, would simply not be added to the next queue. A "face detection" thread can stay alive and perform the following: 1. Grab a single frame or a batch of frames from "face detection" queue 2. Run detector (opencv can run our NN faster than tensorflow) 3. For each face, add to "face feature generation" queue A "face feature generation" thread can stay alive and perform the following: 1. Grab a single face or a batch of faces from "face feature generation" queue 2. Run feature generator (need to find pose-invariant generator) 3. For each face, add to "final processing" queue Processing Steps 5 and 6 can be combined and thought of as final processing. Multiprocessing workers can pull faces from the "final processing" queue and then run steps 5, 6, and 7. We are looking for someone with extensive experience working with image/video processing pipelines and machine learning algorithms. The ideal candidate will have a significant amount of time to devote to this project per week so that we can move forward swiftly.