Functions to accelerate activation functions in Synet Framework. More...

Functions
SIMD_API void	SimdSynetElu32f (const float src, size_t size, const float alpha, float *dst)
	Calculates ELU activation for an FP32 array. More...

SIMD_API void	SimdSynetGelu32f (const float src, size_t size, float dst)
	Calculates exact GELU activation for an FP32 array. More...

SIMD_API void	SimdSynetHardSigmoid32f (const float src, size_t size, const float scale, const float shift, float dst)
	Calculates HardSigmoid activation for an FP32 array. More...

SIMD_API void	SimdSynetHswish32f (const float src, size_t size, const float shift, const float scale, float dst)
	Calculates H-Swish activation for an FP32 array. More...

SIMD_API void	SimdSynetMish32f (const float src, size_t size, const float threshold, float *dst)
	Calculates Mish activation function (https://arxiv.org/abs/1908.08681) for an FP32 array. More...

SIMD_API void	SimdSynetPreluLayerForward (const float src, const float slope, size_t channels, size_t spatial, float *dst, SimdTensorFormatType format)
	Performs PReLU activation with one slope per channel for an FP32 tensor. More...

SIMD_API void	SimdSynetRelu32f (const float src, size_t size, const float slope, float *dst)
	Calculates leaky ReLU function for FP32 array. More...

SIMD_API void	SimdSynetRelu16b (const uint16_t src, size_t size, const float slope, uint16_t *dst)
	Calculates leaky ReLU function for BF16 array. More...

SIMD_API void	SimdSynetRestrictRange32f (const float src, size_t size, const float lower, const float upper, float dst)
	Clamps FP32 array values to a given range. More...

SIMD_API void	SimdSynetSigmoid32f (const float src, size_t size, const float slope, float *dst)
	Calculates sigmoid function for FP32 array. More...

SIMD_API void	SimdSynetSoftplus32f (const float src, size_t size, const float beta, const float threshold, float dst)
	Applies Softplus activation to a 32-bit floating point array. More...

SIMD_API void	SimdSynetSwish32f (const float src, size_t size, const float slope, float *dst)
	Applies Swish activation to a 32-bit floating point array. More...

SIMD_API void	SimdSynetTanh32f (const float src, size_t size, const float slope, float *dst)
	Applies hyperbolic tangent activation to a 32-bit floating point array. More...

Detailed Description

Functions to accelerate activation functions in Synet Framework.

Function Documentation

◆ SimdSynetElu32f()

void SimdSynetElu32f	(	const float *	src,
		size_t	size,
		const float *	alpha,
		float *	dst
	)

Calculates ELU activation for an FP32 array.

The input and output arrays must have the same size.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = src[i] >= 0 ? src[i] : alpha[0]*(exp(src[i]) - 1);

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 array.
[in]	size	- a number of elements in the input and output arrays.
[in]	alpha	- a pointer to ELU alpha parameter. Only alpha[0] is used.
[out]	dst	- a pointer to the output FP32 array.

◆ SimdSynetGelu32f()

void SimdSynetGelu32f	(	const float *	src,
		size_t	size,
		float *	dst
	)

Calculates exact GELU activation for an FP32 array.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = src[i] * (1 + erf(src[i]/sqrt(2))) / 2;

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 array.
[in]	size	- a number of elements in the input and output arrays.
[out]	dst	- a pointer to the output FP32 array.

◆ SimdSynetHardSigmoid32f()

void SimdSynetHardSigmoid32f	(	const float *	src,
		size_t	size,
		const float *	scale,
		const float *	shift,
		float *	dst
	)

Calculates HardSigmoid activation for an FP32 array.

The input and output arrays must have the same size.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = Max(0, Min(src[i] * scale[0] + shift[0], 1));

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 array.
[in]	size	- a number of elements in the input and output arrays.
[in]	scale	- a pointer to scale parameter. Only scale[0] is used. This parameter is equal to 1/6 in PyTorch documentation.
[in]	shift	- a pointer to shift parameter. Only shift[0] is used. This parameter is equal to 1/2 in PyTorch documentation.
[out]	dst	- a pointer to the output FP32 array.

◆ SimdSynetHswish32f()

void SimdSynetHswish32f	(	const float *	src,
		size_t	size,
		const float *	shift,
		const float *	scale,
		float *	dst
	)

Calculates H-Swish activation for an FP32 array.

The input and output arrays must have the same size.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = Max(Min(src[i], shift[0]) + shift[0], 0)*scale[0]*src[i];

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 array.
[in]	size	- a number of elements in the input and output arrays.
[in]	shift	- a pointer to shift parameter. Only shift[0] is used. It is equal to 3 in the original paper.
[in]	scale	- a pointer to scale parameter. Only scale[0] is used. It is equal to 1/6 in the original paper.
[out]	dst	- a pointer to the output FP32 array.

◆ SimdSynetMish32f()

void SimdSynetMish32f	(	const float *	src,
		size_t	size,
		const float *	threshold,
		float *	dst
	)

Calculates Mish activation function (https://arxiv.org/abs/1908.08681) for an FP32 array.

The function uses threshold[0] as an overflow guard: values greater than the threshold are copied to the destination because Mish(x) approaches x for large positive x.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = src[i] > threshold[0] ? src[i] : src[i] * Tanh(Log(Exp(src[i]) + 1));

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 array.
[in]	size	- a size of input and output arrays.
[in]	threshold	- a pointer to one FP32 threshold parameter.
[out]	dst	- a pointer to the output FP32 array.

◆ SimdSynetPreluLayerForward()

void SimdSynetPreluLayerForward	(	const float *	src,
		const float *	slope,
		size_t	channels,
		size_t	spatial,
		float *	dst,
		SimdTensorFormatType	format
	)

Performs PReLU activation with one slope per channel for an FP32 tensor.

The function supports SimdTensorFormatNchw and SimdTensorFormatNhwc. For each element it keeps positive values unchanged and multiplies negative values by the slope of the corresponding channel.

Algorithm's details (example for NCHW tensor format):

for(c = 0; c < channels; ++c)
    for(s = 0; s < spatial; ++s)
        dst[c*spatial + s] = src[c*spatial + s] > 0 ? src[c*spatial + s] : slope[c]*src[c*spatial + s];

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input FP32 tensor. The size of the array must be equal to channels*spatial.
[in]	slope	- a pointer to per-channel FP32 slope coefficients. The size of the array must be equal to channels.
[in]	channels	- a number of input and output tensor channels.
[in]	spatial	- a spatial size (height*width) of input and output tensor.
[out]	dst	- a pointer to the output FP32 tensor. The size of the array must be equal to channels*spatial.
[in]	format	- a format of input and output tensor. It can be SimdTensorFormatNchw or SimdTensorFormatNhwc.

◆ SimdSynetRelu32f()

void SimdSynetRelu32f	(	const float *	src,
		size_t	size,
		const float *	slope,
		float *	dst
	)

Calculates leaky ReLU function for FP32 array.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = Max(0, src[i]) + slope[0]*Min(0, src[i]);

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	slope	- a pointer to slope parameter for negative values.
[out]	dst	- a pointer to output 32-bit float array.

◆ SimdSynetRelu16b()

void SimdSynetRelu16b	(	const uint16_t *	src,
		size_t	size,
		const float *	slope,
		uint16_t *	dst
	)

Calculates leaky ReLU function for BF16 array.

Algorithm's details:

for(i = 0; i < size; ++i)
{
    value = BFloat16ToFloat32(src[i]);
    dst[i] = Float32ToBFloat16(Max(0, value) + slope[0]*Min(0, value));
}

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 16-bit brain-float array.
[in]	size	- a size of input and output arrays.
[in]	slope	- a pointer to slope parameter for negative values.
[out]	dst	- a pointer to output 16-bit brain-float array.

◆ SimdSynetRestrictRange32f()

void SimdSynetRestrictRange32f	(	const float *	src,
		size_t	size,
		const float *	lower,
		const float *	upper,
		float *	dst
	)

Clamps FP32 array values to a given range.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = Min(Max(src[i], lower[0]), upper[0]);

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	lower	- a pointer to lower bound.
[in]	upper	- a pointer to upper bound.
[out]	dst	- a pointer to the output 32-bit float array.

◆ SimdSynetSigmoid32f()

void SimdSynetSigmoid32f	(	const float *	src,
		size_t	size,
		const float *	slope,
		float *	dst
	)

Calculates sigmoid function for FP32 array.

Algorithm's details:

for(i = 0; i < size; ++i)
    dst[i] = 1/(1 + exp(-slope[0]*src[i]));

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	slope	- a pointer to slope parameter.
[out]	dst	- a pointer to output 32-bit float array.

◆ SimdSynetSoftplus32f()

void SimdSynetSoftplus32f	(	const float *	src,
		size_t	size,
		const float *	beta,
		const float *	threshold,
		float *	dst
	)

Applies Softplus activation to a 32-bit floating point array.

Algorithm's details:

_beta = beta[0];
_threshold = threshold[0];
for(i = 0; i < size; ++i)
{
    value = src[i];
    dst[i] = value > _threshold ? value : log(1 + exp(value*_beta))/_beta;
}

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	beta	- a pointer to the Softplus beta scalar parameter.
[in]	threshold	- a pointer to the Softplus threshold scalar parameter. If input value is greater than threshold then output is equal to input.
[out]	dst	- a pointer to the output 32-bit float array.

◆ SimdSynetSwish32f()

void SimdSynetSwish32f	(	const float *	src,
		size_t	size,
		const float *	slope,
		float *	dst
	)

Applies Swish activation to a 32-bit floating point array.

Algorithm's details:

_slope = slope[0];
for(i = 0; i < size; ++i)
    dst[i] = src[i]/(1 + exp(-_slope*src[i]));

Note: This function is used in Synet Framework.

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	slope	- a pointer to the Swish slope scalar parameter.
[out]	dst	- a pointer to the output 32-bit float array.

◆ SimdSynetTanh32f()

void SimdSynetTanh32f	(	const float *	src,
		size_t	size,
		const float *	slope,
		float *	dst
	)

Applies hyperbolic tangent activation to a 32-bit floating point array.

Note: This function is used in Synet Framework.

Algorithm's details:

_slope = slope[0];
for(i = 0; i < size; ++i)
    dst[i] = tanh(src[i]*_slope);

Parameters

[in]	src	- a pointer to the input 32-bit float array.
[in]	size	- a size of input and output arrays.
[in]	slope	- a pointer to the Tanh slope scalar parameter.
[out]	dst	- a pointer to the output 32-bit float array.

Simd Library Documentation.

Functions

Detailed Description

Function Documentation

◆ SimdSynetElu32f()

◆ SimdSynetGelu32f()

◆ SimdSynetHardSigmoid32f()

◆ SimdSynetHswish32f()

◆ SimdSynetMish32f()

◆ SimdSynetPreluLayerForward()

◆ SimdSynetRelu32f()

◆ SimdSynetRelu16b()

◆ SimdSynetRestrictRange32f()

◆ SimdSynetSigmoid32f()

◆ SimdSynetSoftplus32f()

◆ SimdSynetSwish32f()

◆ SimdSynetTanh32f()