SynetQuantizedMergedConvolution Class Reference

The SynetQuantizedMergedConvolution class is a C++ wrapper of UINT8 quantized merged convolution. More...

#include <SimdSynet.hpp>

Public Member Functions
	SynetQuantizedMergedConvolution ()
virtual	~SynetQuantizedMergedConvolution ()
SIMD_INLINE void	Init (size_t batch, const SimdConvolutionParameters *convs, size_t count, int add)
SIMD_INLINE bool	Enable () const
SIMD_INLINE size_t	ExternalBufferSize () const
SIMD_INLINE size_t	InternalBufferSize () const
SIMD_INLINE const char *	Info () const
SIMD_INLINE void	SetParams (const float *ioScale, const uint8_t *ioZero, const int8_t *const *weight, const float *const *weightScale, const int32_t *const *bias)
SIMD_INLINE void	Forward (const uint8_t *src, uint8_t *buf, uint8_t *dst)
SIMD_INLINE void	Clear ()

Detailed Description

The SynetQuantizedMergedConvolution class is a C++ wrapper of UINT8 quantized merged convolution.

The class wraps C API functions SimdSynetQuantizedMergedConvolutionInit, SimdSynetQuantizedMergedConvolutionExternalBufferSize, SimdSynetQuantizedMergedConvolutionInternalBufferSize, SimdSynetQuantizedMergedConvolutionInfo, SimdSynetQuantizedMergedConvolutionSetParams and SimdSynetQuantizedMergedConvolutionForward. It fuses a sequence of two or three NHWC UINT8-to-UINT8 quantized convolutions into one forward call: pointwise + depthwise, depthwise + pointwise, or pointwise + depthwise + pointwise. Source and destination tensors are UINT8, weights are INT8, and each tensor edge has its own scale and zero point. Ordinary convolutions use 1x1 or 3x3 kernels, depthwise convolutions use 3x3, 5x5 or 7x7 kernels; kernels and strides must be square, dilation must be 1 and stride must be 1, 2 or 3. If add is non-zero for a three-convolution chain, the final output is a requantized residual sum of the convolution output and the original input (add = 1 adds output to source, add = 2 adds source to output).

Call Init() and SetParams() before Forward(). Use Enable() to check that a context was created. The context is released by Clear() or by the destructor.

Using example:

#include "Simd/SimdSynet.hpp"

int main()
{
    const size_t batch = 1, srcC = 4, srcH = 8, srcW = 8, midC = 8, count = 2, add = 0;
    SimdConvolutionParameters convs[2] = {};

    convs[0].srcC = srcC;
    convs[0].srcH = srcH;
    convs[0].srcW = srcW;
    convs[0].srcT = SimdTensorData8u;
    convs[0].srcF = SimdTensorFormatNhwc;
    convs[0].dstC = midC;
    convs[0].kernelY = 1;
    convs[0].kernelX = 1;
    convs[0].dilationY = 1;
    convs[0].dilationX = 1;
    convs[0].strideY = 1;
    convs[0].strideX = 1;
    convs[0].padY = 0;
    convs[0].padX = 0;
    convs[0].padH = 0;
    convs[0].padW = 0;
    convs[0].group = 1;
    convs[0].activation = SimdConvolutionActivationIdentity;
    convs[0].dstH = srcH;
    convs[0].dstW = srcW;
    convs[0].dstT = SimdTensorData8u;
    convs[0].dstF = SimdTensorFormatNhwc;

    convs[1].srcC = midC;
    convs[1].srcH = convs[0].dstH;
    convs[1].srcW = convs[0].dstW;
    convs[1].srcT = SimdTensorData8u;
    convs[1].srcF = SimdTensorFormatNhwc;
    convs[1].dstC = midC;
    convs[1].kernelY = 3;
    convs[1].kernelX = 3;
    convs[1].dilationY = 1;
    convs[1].dilationX = 1;
    convs[1].strideY = 1;
    convs[1].strideX = 1;
    convs[1].padY = 1;
    convs[1].padX = 1;
    convs[1].padH = 1;
    convs[1].padW = 1;
    convs[1].group = midC;
    convs[1].activation = SimdConvolutionActivationIdentity;
    convs[1].dstH = convs[1].srcH;
    convs[1].dstW = convs[1].srcW;
    convs[1].dstT = SimdTensorData8u;
    convs[1].dstF = SimdTensorFormatNhwc;

    std::vector<uint8_t> src(batch * srcH * srcW * srcC);
    std::vector<int8_t> weight0(convs[0].kernelY * convs[0].kernelX * convs[0].srcC * convs[0].dstC);
    std::vector<int8_t> weight1(convs[1].kernelY * convs[1].kernelX * convs[1].dstC);
    std::vector<float> weightScale0(convs[0].dstC, 0.02f), weightScale1(convs[1].dstC, 0.03f);
    std::vector<int32_t> bias0(convs[0].dstC, 1), bias1(convs[1].dstC, 2);
    float ioScale[3] = { 0.01f, 0.015f, 0.02f };
    uint8_t ioZero[3] = { 128, 127, 126 };
    std::vector<uint8_t> dst(batch * convs[1].dstH * convs[1].dstW * convs[1].dstC, 0);
    const int8_t * weight[2] = { weight0.data(), weight1.data() };
    const float * weightScale[2] = { weightScale0.data(), weightScale1.data() };
    const int32_t * bias[2] = { bias0.data(), bias1.data() };
    for (size_t i = 0; i < src.size(); ++i)
        src[i] = uint8_t(i);
    for (size_t i = 0; i < weight0.size(); ++i)
        weight0[i] = int8_t(i);
    for (size_t i = 0; i < weight1.size(); ++i)
        weight1[i] = int8_t(i);

    Simd::SynetQuantizedMergedConvolution mergedConvolution;
    mergedConvolution.Init(batch, convs, count, add);
    if (mergedConvolution.Enable())
    {
        mergedConvolution.SetParams(ioScale, ioZero, weight, weightScale, bias);
        mergedConvolution.Forward(src.data(), NULL, dst.data());
    }

    return 0;
}

Constructor & Destructor Documentation

SynetQuantizedMergedConvolution()

SynetQuantizedMergedConvolution

(

)

Creates a new empty SynetQuantizedMergedConvolution class.

~SynetQuantizedMergedConvolution()

virtual ~SynetQuantizedMergedConvolution ( )

virtual

SynetQuantizedMergedConvolution class destructor. Releases internal context.

Member Function Documentation

Init()

SIMD_INLINE void Init	(	size_t	batch,
		const SimdConvolutionParameters *	convs,
		size_t	count,
		int	add
	)

Initializes (or re-initializes) a quantized merged convolution context.

Creates an internal context with using of function SimdSynetQuantizedMergedConvolutionInit. The context is recreated only if batch size, convolution count, residual-add mode or any convolution parameters were changed.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionInit.

Parameters

[in]	batch	- a batch size.
[in]	convs	- an array with convolution parameters in execution order.
[in]	count	- a number of merged convolutions. It must be 2 or 3.
[in]	add	- a residual addition mode: 0 disables addition, 1 adds output to source, 2 adds source to output.

Enable()

SIMD_INLINE bool Enable

(

)

const

Checks that the internal merged convolution context was created.

Returns

true if the context exists and Forward() can be called.

ExternalBufferSize()

SIMD_INLINE size_t ExternalBufferSize

(

)

const

Gets the size in bytes of caller-provided temporary buffer for quantized merged convolution.

The returned value is a number of bytes. It depends on the implementation selected during initialization and can be used when allocating the buf argument of Forward(). Some implementations return 1 when they do not need external temporary storage.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionExternalBufferSize.

Returns

a number of bytes required for external temporary buffer.

InternalBufferSize()

SIMD_INLINE size_t InternalBufferSize

(

)

const

Gets the size in bytes of internal storage used by the quantized merged convolution context.

The returned value reports internal temporary buffers, reordered weights, biases, norms, zero points and an optional fallback temporary buffer already allocated by the context.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionInternalBufferSize.

Returns

a number of bytes used by internal buffers.

Info()

SIMD_INLINE const char * Info

(

)

const

Gets a short description of the selected quantized merged convolution implementation.

The returned string contains the implementation extension and algorithm name. The returned pointer is owned by the context and remains valid until the next call of this function or until the context is released.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionInfo.

Returns

a string with description of internal implementation. NULL if the context was not created.

SetParams()

SIMD_INLINE void SetParams	(	const float *	ioScale,
		const uint8_t *	ioZero,
		const int8_t *const *	weight,
		const float *const *	weightScale,
		const int32_t *const *	bias
	)

Sets INT8 weights, INT32 biases and quantization parameters for quantized merged convolution.

This function must be called before Forward(). Arrays weight, weightScale and bias contain one pointer per merged convolution. The ioScale and ioZero arrays contain quantization parameters for every edge between convolutions: input, intermediate outputs and final output. When residual addition is enabled, one additional scale and zero point are used for the residual-sum output. Individual bias pointers can be NULL.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionSetParams.

Parameters

[in]	ioScale	- a pointer to FP32 input/intermediate/output tensor scales.
[in]	ioZero	- a pointer to UINT8 input/intermediate/output tensor zero points.
[in]	weight	- an array of pointers to INT8 convolution weights. The array size must be equal to the number of merged convolutions.
[in]	weightScale	- an array of pointers to per-output-channel FP32 weight scales. The array size must be equal to the number of merged convolutions.
[in]	bias	- an array of pointers to per-output-channel INT32 biases. The array size must be equal to the number of merged convolutions. Individual pointers can be NULL.

Forward()

SIMD_INLINE void Forward	(	const uint8_t *	src,
		uint8_t *	buf,
		uint8_t *	dst
	)

Performs quantized merged convolution forward propagation.

The function applies the fused UINT8 convolution sequence stored in the context created by Init() and SetParams(). The buf argument can be NULL (it causes usage of internal buffer). If Init() was called with a non-zero add mode, the source tensor is combined with the convolution output and the result is requantized to UINT8.

Note

This function is a C++ wrapper for function SimdSynetQuantizedMergedConvolutionForward.

Parameters

[in]	src	- a pointer to UINT8 input tensor of the first convolution.
[out]	buf	- a pointer to an external temporary byte buffer. Can be NULL.
[out]	dst	- a pointer to UINT8 output tensor of the last convolution or residual sum.

Clear()

SIMD_INLINE void Clear

(

)

Releases internal context and clears stored merged convolution parameters.