pyVHR.extraction package¶

Submodules¶

pyVHR.extraction.sig_extraction_methods module¶

class pyVHR.extraction.sig_extraction_methods.SignalProcessingParams¶

Bases: object

This class contains usefull parameters used by this module.

RGB_LOW_TH (numpy.int32): RGB low-threshold value.

RGB_HIGH_TH (numpy.int32): RGB high-threshold value.

RGB_HIGH_TH = 200¶

RGB_LOW_TH = 55¶

pyVHR.extraction.sig_extraction_methods.holistic_mean(im, RGB_LOW_TH, RGB_HIGH_TH)¶

This method computes the RGB-Mean Signal excluding ‘im’ pixels that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

Parameters

im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
RGB_LOW_TH (numpy.int32) – RGB low threshold value.
RGB_HIGH_TH (numpy.int32) – RGB high threshold value.

Returns

RGB-Mean Signal as float32 ndarray with shape [1,3], where 1 is the single estimator, and 3 are r-mean, g-mean and b-mean.

pyVHR.extraction.sig_extraction_methods.landmarks_mean(ldmks, im, square, RGB_LOW_TH, RGB_HIGH_TH)¶

This method computes the RGB-Mean Signal excluding ‘im’ pixels that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

Parameters

ldmks (float32 ndarray) – landmakrs as ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean (value is not important), g-mean (value is not important), b-mean (value is not important).
im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
square (numpy.float32) – side size of square patches.
RGB_LOW_TH (numpy.int32) – RGB low threshold value.
RGB_HIGH_TH (numpy.int32) – RGB high threshold value.

Returns

RGB-Mean Signal as float32 ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean, g-mean, b-mean.

pyVHR.extraction.sig_extraction_methods.landmarks_mean_custom_rect(ldmks, im, rects, RGB_LOW_TH, RGB_HIGH_TH)¶

This method computes the RGB-Mean Signal excluding ‘im’ pixels that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

Parameters

ldmks (float32 ndarray) – landmakrs as ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean (value is not important), g-mean (value is not important), b-mean (value is not important).
im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
rects (float32 ndarray) – positive float32 np.ndarray of shape [num_landmarks, 2]. If the list of used landmarks is [1,2,3] and rects_dim is [[10,20],[12,13],[40,40]] then the landmark number 2 will have a rectangular patch of xy-dimension 12x13.
RGB_LOW_TH (numpy.int32) – RGB low threshold value.
RGB_HIGH_TH (numpy.int32) – RGB high threshold value.

Returns

RGB-Mean Signal as float32 ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean, g-mean, b-mean.

pyVHR.extraction.sig_extraction_methods.landmarks_median(ldmks, im, square, RGB_LOW_TH, RGB_HIGH_TH)¶

This method computes the RGB-Median Signal excluding ‘im’ pixels that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

Parameters

ldmks (float32 ndarray) – landmakrs as ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean (value is not important), g-mean (value is not important), b-mean (value is not important).
im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
square (numpy.float32) – side size of square patches.
RGB_LOW_TH (numpy.int32) – RGB low threshold value.
RGB_HIGH_TH (numpy.int32) – RGB high threshold value.

Returns

RGB-Median Signal as float32 ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean, g-mean, b-mean.

pyVHR.extraction.sig_extraction_methods.landmarks_median_custom_rect(ldmks, im, rects, RGB_LOW_TH, RGB_HIGH_TH)¶

This method computes the RGB-Median Signal excluding ‘im’ pixels that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

Parameters

ldmks (float32 ndarray) – landmakrs as ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean (value is not important), g-mean (value is not important), b-mean (value is not important).
im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
rects (float32 ndarray) – positive float32 np.ndarray of shape [num_landmarks, 2]. If the list of used landmarks is [1,2,3] and rects_dim is [[10,20],[12,13],[40,40]] then the landmark number 2 will have a rectangular patch of xy-dimension 12x13.
RGB_LOW_TH (numpy.int32) – RGB low threshold value.
RGB_HIGH_TH (numpy.int32) – RGB high threshold value.

Returns

RGB-Median Signal as float32 ndarray with shape [num_landmarks, 5], where the second dimension contains y-coord, x-coord, r-mean, g-mean, b-mean.

pyVHR.extraction.sig_processing module¶

class pyVHR.extraction.sig_processing.SignalProcessing¶

Bases: object

This class performs offline signal extraction with different methods:

holistic.
squared / rectangular patches.

choose_cuda_device(n)¶

Choose a CUDA device.

Parameters: n (int) – number of a CUDA device.

display_cuda_device()¶: Display your CUDA devices.

extract_holistic(videoFileName)¶

This method compute the RGB-mean signal using the whole skin (holistic);

Parameters: videoFileName (str) – video file name or path.
Returns: RGB signal as ndarray with shape [num_frames, 1, rgb_channels]. The second dimension is 1 because the whole skin is considered as one estimators.
Return type: float32 ndarray

extract_patches(videoFileName, region_type, sig_extraction_method)¶

This method compute the RGB-mean signal using specific skin regions (patches).

Parameters

videoFileName (str) – video file name or path.
region_type (str) – patches types can be “squares” or “rects”.
sig_extraction_method (str) – RGB signal can be computed with “mean” or “median”. We recommend to use mean.

Returns

RGB signal as ndarray with shape [num_frames, num_patches, rgb_channels].

Return type

float32 ndarray

extract_raw_holistic(videoFileName)¶

Locates the skin pixels in each frame. This method is intended for rPPG methods that use raw video signal.

Parameters: videoFileName (str) – video file name or path.
Returns: raw signal as float32 ndarray with shape [num_frames, rows, columns, rgb_channels].
Return type: float32 ndarray

get_visualize_patches()¶

Get the ‘skin+patches’ images produced by the last processing. Remember to set pyVHR.extraction.sig_processing.SignalProcessing.set_visualize_skin_and_landmarks() correctly.

Returns: list of cv2 images; each image is a ndarray with shape [rows, columns, rgb_channels].
Return type: list of ndarray

get_visualize_skin()¶

Get the skin images produced by the last processing. Remember to set pyVHR.extraction.sig_processing.SignalProcessing.set_visualize_skin_and_landmarks() correctly.

Returns: list of cv2 images; each image is a ndarray with shape [rows, columns, rgb_channels].
Return type: list of ndarray

set_landmarks(landmarks_list)¶

Set the patches centers (landmarks) that will be used for signal processing. There are 468 facial points you can choose; for visualizing their identification number please use pyVHR.plot.visualize.visualize_landmarks_list().

Parameters: landmarks_list (list) – list of positive integers between 0 and 467 that identify patches centers (landmarks).

set_rect_patches_sides(rects_dim)¶

Set the dimension of each rectangular patch. There are 468 facial points you can choose; for visualizing their identification number please use pyVHR.plot.visualize.visualize_landmarks_list().

Parameters: rects_dim (float32 ndarray) – positive float32 np.ndarray of shape [num_landmarks, 2]. If the list of used landmarks is [1,2,3] and rects_dim is [[10,20],[12,13],[40,40]] then the landmark number 2 will have a rectangular patch of xy-dimension 12x13.

set_skin_extractor(extractor)¶

Set the skin extractor that will be used for skin extraction.

Parameters: extractor – instance of a skin_extraction class (see pyVHR.extraction.skin_extraction_methods).

set_square_patches_side(square_side)¶

Set the dimension of the square patches that will be used for signal processing. There are 468 facial points you can choose; for visualizing their identification number please use pyVHR.plot.visualize.visualize_landmarks_list().

Parameters: square_side (float) – positive float that defines the length of the square patches.

set_total_frames(n)¶

Set the total frames to be processed; if you want to process all the possible frames use n = 0.

Parameters: n (int) – number of frames to be processed.

set_visualize_skin_and_landmarks(visualize_skin=False, visualize_landmarks=False, visualize_landmarks_number=False, visualize_patch=False)¶

Set visualization parameters. You can retrieve visualization output with the methods pyVHR.extraction.sig_processing.SignalProcessing.get_visualize_skin() and pyVHR.extraction.sig_processing.SignalProcessing.get_visualize_patches()

Parameters

visualize_skin (bool) – The skin and the patches will be visualized.
visualize_landmarks (bool) – The landmarks (centers of patches) will be visualized.
visualize_landmarks_number (bool) – The landmarks number will be visualized.
visualize_patch (bool) – The patches outline will be visualized.

pyVHR.extraction.skin_extraction_methods module¶

class pyVHR.extraction.skin_extraction_methods.SkinExtractionConvexHull(device='CPU')¶

Bases: object

This class performs skin extraction on CPU/GPU using a Convex Hull segmentation obtained from facial landmarks.

extract_skin(image, ldmks)¶

This method extract the skin from an image using Convex Hull segmentation.

Parameters

image (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
ldmks (float32 ndarray) – landmarks used to create the Convex Hull; ldmks is a ndarray with shape [num_landmarks, xy_coordinates].

Returns

Cropped skin-image and non-cropped skin-image; both are uint8 ndarray with shape [rows, columns, rgb_channels].

class pyVHR.extraction.skin_extraction_methods.SkinExtractionFaceParsing(device='CPU')¶

Bases: object

This class performs skin extraction on CPU/GPU using Face Parsing. https://github.com/zllrunning/face-parsing.PyTorch

extract_skin(image, ldmks)¶

This method extract the skin from an image using Face Parsing. Landmarks (ldmks) are used to create a facial bounding box for cropping the face; this way the network used in Face Parsing is more accurate.

Parameters

image (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].
ldmks (float32 ndarray) – ndarray with shape [num_landmarks, xy_coordinates].

Returns

Cropped skin-image and non-cropped skin-image; both are uint8 ndarray with shape [rows, columns, rgb_channels].

extraction(im, nda_im)¶

This method performs skin extraction using Face Parsing.

Parameters

im (torch.Tensor) – torch.Tensor with size [rows, columns, rgb_channels]
nda_im (uint8 ndarray) – ndarray with shape [rows, columns, rgb_channels].

Returns

skin-image as uint8 ndarray with shape [rows, columns, rgb_channels].

class pyVHR.extraction.skin_extraction_methods.SkinProcessingParams¶

Bases: object

This class contains usefull parameters used by this module.

RGB_LOW_TH (numpy.int32): RGB low-threshold value.

RGB_HIGH_TH (numpy.int32): RGB high-threshold value.

RGB_HIGH_TH = 200¶

RGB_LOW_TH = 55¶

pyVHR.extraction.skin_extraction_methods.bbox2_CPU(img)¶

Parameters: img (ndarray) – ndarray with shape [rows, columns, rgb_channels].
Returns: Four cropping coordinates (row, row, column, column) for removing black borders (RGB [O,O,O]) from img.

pyVHR.extraction.skin_extraction_methods.bbox2_GPU(img)¶

Parameters: img (cupy.ndarray) – cupy.ndarray with shape [rows, columns, rgb_channels].
Returns: Four cropping coordinates (row, row, column, column) for removing black borders (RGB [O,O,O]) from img. The returned variables are on GPU.

pyVHR.extraction.skin_extraction_methods.kernel_cuda_skin_copy_and_filter()¶

Return a Numba cuda.jit kernel defined as:

@cuda.jit(‘void(uint8[:,:,:], int32[:,:], uint8[:,:,:], int32[:])’) def __kernel_cuda_skin_copy_and_filter(orig, pars, new, low_high_filter):

‘’ This method removes pixels from the image ‘orig’ that are not skin, or that are outside the RGB range [low_high_filter[0], low_high_filter[1]] (extremes are included). ‘’

This method is important for users who do not use a GPU, beacause they can’t compile @cuda.jit.

pyVHR.extraction.skin_extraction_methods.kernel_skin_copy_and_filter(orig, pars, RGB_LOW_TH, RGB_HIGH_TH)¶: This method removes pixels from the image ‘orig’ that are not skin, or that are outside the RGB range [RGB_LOW_TH, RGB_HIGH_TH] (extremes are included).

pyVHR.extraction.utils module¶

class pyVHR.extraction.utils.MagicLandmarks¶

Bases: object

This class contains usefull lists of landmarks identification numbers.

cheek_left_bottom = [215, 138, 135, 210, 212, 57, 216, 207, 192]¶

cheek_left_top = [116, 111, 117, 118, 119, 100, 47, 126, 101, 123, 137, 177, 50, 36, 209, 129, 205, 147, 177, 215, 187, 207, 206, 203]¶

cheek_right_bottom = [435, 427, 416, 364, 394, 422, 287, 410, 434, 436]¶

cheek_right_top = [349, 348, 347, 346, 345, 447, 323, 280, 352, 330, 371, 358, 423, 426, 425, 427, 411, 376]¶

chin = [204, 170, 140, 194, 201, 171, 175, 200, 418, 396, 369, 421, 431, 379, 424]¶

equispaced_facial_points = [2, 3, 4, 5, 6, 8, 9, 10, 18, 21, 32, 35, 36, 43, 46, 47, 48, 50, 54, 58, 67, 68, 69, 71, 92, 93, 101, 103, 104, 108, 109, 116, 117, 118, 123, 132, 134, 135, 138, 139, 142, 148, 149, 150, 151, 152, 182, 187, 188, 193, 197, 201, 205, 206, 207, 210, 211, 212, 216, 234, 248, 251, 262, 265, 266, 273, 277, 278, 280, 284, 288, 297, 299, 322, 323, 330, 332, 333, 337, 338, 345, 346, 361, 363, 364, 367, 368, 371, 377, 379, 411, 412, 417, 421, 425, 426, 427, 430, 432, 436]¶

eye_left = [127, 234, 34, 139, 70, 53, 124, 35, 111, 228, 230, 121, 244, 189, 222, 143]¶

eye_right = [283, 300, 368, 353, 264, 372, 454, 340, 448, 450, 452, 464, 417, 441, 444, 282, 276, 446, 368]¶

forehead_center = [10, 151, 9, 8, 107, 336, 285, 55, 8]¶

forehead_left = [21, 71, 68, 54, 103, 104, 63, 70, 53, 52, 65, 107, 66, 108, 69, 67, 109, 105]¶

forehoead_right = [338, 337, 336, 296, 285, 295, 282, 334, 293, 301, 251, 298, 333, 299, 297, 332, 284]¶

high_prio_forehead = [10, 67, 69, 104, 108, 109, 151, 299, 337, 338]¶

high_prio_left_cheek = [36, 47, 50, 100, 101, 116, 117, 118, 119, 123, 126, 147, 187, 203, 205, 206, 207, 216]¶

high_prio_nose = [3, 4, 5, 6, 45, 51, 115, 122, 131, 134, 142, 174, 195, 196, 197, 198, 209, 217, 220, 236, 248, 275, 277, 281, 360, 363, 399, 419, 420, 429, 437, 440, 456]¶

high_prio_right_cheek = [266, 280, 329, 330, 346, 347, 347, 348, 355, 371, 411, 423, 425, 426, 427, 436]¶

left_eye = [157, 144, 145, 22, 23, 25, 154, 31, 160, 33, 46, 52, 53, 55, 56, 189, 190, 63, 65, 66, 70, 221, 222, 223, 225, 226, 228, 229, 230, 231, 232, 105, 233, 107, 243, 124]¶

mid_prio_chin = [135, 138, 169, 170, 199, 208, 210, 211, 214, 262, 288, 416, 428, 430, 431, 432, 433, 434]¶

mid_prio_forehead = [8, 9, 21, 68, 103, 251, 284, 297, 298, 301, 332, 333, 372, 383]¶

mid_prio_left_cheek = [34, 111, 137, 156, 177, 192, 213, 227, 234]¶

mid_prio_mouth = [92, 164, 165, 167, 186, 212, 322, 391, 393, 410]¶

mid_prio_nose = [1, 44, 49, 114, 120, 121, 128, 168, 188, 351, 358, 412]¶

mid_prio_right_cheek = [340, 345, 352, 361, 454]¶

mounth = [391, 393, 11, 269, 270, 271, 287, 164, 165, 37, 167, 40, 43, 181, 313, 314, 186, 57, 315, 61, 321, 73, 76, 335, 83, 85, 90, 106]¶

mounth_down = [43, 106, 83, 18, 406, 335, 273, 424, 313, 194, 204]¶

mounth_up = [186, 92, 167, 393, 322, 410, 287, 39, 269, 61, 164]¶

nose = [193, 417, 168, 188, 6, 412, 197, 174, 399, 456, 195, 236, 131, 51, 281, 360, 440, 4, 220, 219, 305]¶

right_eye = [384, 385, 386, 259, 388, 261, 265, 398, 276, 282, 283, 285, 413, 293, 296, 300, 441, 442, 445, 446, 449, 451, 334, 463, 336, 464, 467, 339, 341, 342, 353, 381, 373, 249, 253, 255]¶

pyVHR.extraction.utils.draw_rects(image, xcenters, ycenters, xsides, ysides, color)¶: This method is used to draw N rectangles on a image.

pyVHR.extraction.utils.extract_frames_yield(videoFileName)¶: This method yield the frames of a video file name or path.

pyVHR.extraction.utils.get_fps(videoFileName)¶: This method returns the fps of a video file name or path.

pyVHR.extraction.utils.get_magic_landmarks()¶: returns high_priority and mid_priority list of landmarks identification number

pyVHR.extraction.utils.raw_windowing(raw_signal, wsize, stride, fps)¶

This method is used to divide a Raw signal into overlapping windows.

Parameters

sig (float32 ndarray) – ndarray of images with shape [num_frames, rows, columns, rgb_channels].
wsize (float) – window size in seconds.
stride (float) – stride between overlapping windows in seconds.
fps (float) – frames per seconds.

Returns

windowed signal as a list of length num_windows of float32 ndarray with shape [num_frames, rows, columns, rgb_channels], and a 1D ndarray of times in seconds,where each one is the center of a window.

pyVHR.extraction.utils.sig_windowing(sig, wsize, stride, fps)¶

This method is used to divide a RGB signal into overlapping windows.

Parameters

sig (float32 ndarray) – ndarray with shape [num_frames, num_estimators, rgb_channels].
wsize (float) – window size in seconds.
stride (float) – stride between overlapping windows in seconds.
fps (float) – frames per seconds.

Returns

windowed signal as a list of length num_windows of float32 ndarray with shape [num_estimators, rgb_channels, window_frames], and a 1D ndarray of times in seconds,where each one is the center of a window.

pyVHR.extraction.utils.sliding_straded_win_offline(N, wsize, stride, fps)¶

This method is used to compute all the info for creating an overlapping windows signal.

Parameters

N (int) – length of the signal.
wsize (float) – window size in seconds.
stride (float) – stride between overlapping windows in seconds.
fps (float) – frames per seconds.

Returns

List of ranges, each one contains the indices of a window, and a 1D ndarray of times in seconds, where each one is the center of a window.

pyVHR.datasets package pyVHR.plot package