1.

INTRODUCTION

PSC (Putonghua Shuiping Ceshi) is the only authoritative test for the grading of Mandarin proficiency in China. The form of the test is an oral examination, and the test content is divided into four questions. The first three questions are reading the given characters, words, and articles respectively, which belong to the text-dependent oral evaluation. The first three questions have been successfully scored by machine, and the machine scoring is very close to the manual scoring. The fourth question belongs to Topic Talk. According to the topic determined by the lottery, candidates can express freely and improvise within a time limit of four minutes. Since the content expressed by the candidates cannot be accurately known in advance, the Topic Talk is a text-independent oral evaluation task. Limited to the existing research work on text-independent oral evaluation, manual scoring is still used for this task at present, which undoubtedly increases the manpower and workload of PSC.

Log posterior probability is recognized as the most important feature to measure pronunciation quality in oral evaluation tasks. It has been widely and maturely applied in the text-dependent oral evaluation task of PSC. Ge et al.¹ proposed a posterior probability algorithm based on phoneme confusion expansion network, which can significantly improve the calculation speed of posterior probability without changing the computational complexity of the system. Chen et al.² introduced the knowledge of Mandarin pronunciation linguistics into the posterior probability algorithm, and improved the pronunciation quality evaluation algorithm from the perspective of the phoneme scoring model, which can significantly improve the correlation between manual scoring and machine scoring. Chen et al.³ optimized the probability space of the current computer-aided PSC system from the perspective of phonetics, which not only reduced the confusion caused by the probability space, but also significantly shortened the computing time of the system. Since there is no pre-set reference text in the Topic Talk evaluation of PSC, in order to realize the intelligent evaluation of this item, it is necessary to perform continuous speech flow recognition with a recognizer in advance, and take the optimal recognition result as the reference text of log posteriori probability. In this paper, we introduce a deep neural network to construct an intelligent evaluation model for the Topic Talk in PSC.

2.

INTELLIGENT EVALUATION MODEL OF TOPIC TALK

The total score of Topic Talk item in PSC is 40 points. According to the requirements of the PSC official examination syllabus, the assessors should make a comprehensive evaluation from four aspects: the level of pronunciation standard (25 points), the degree of vocabulary and grammar standard (10 points), natural fluency (5 points), and the appropriate deduction of points if the effective time of expression is less than 3 minutes⁴. This item not only assesses the degree of standard and fluency of candidates’ pronunciation, but also pays attention to the use of vocabulary and grammar. In the Topic Talk evaluation, since the machine cannot know the text expressed by the candidate in advance, the method of forcibly aligning the text with the model in the text-dependent evaluation task cannot be directly adopted. Therefore, it is necessary to add a complete large vocabulary continuous speech recognition module (LVCSR) at the front end of the intelligent evaluation model of Topic Talk item. The intelligent evaluation process of Topic Talk in Mandarin proficiency test is shown in Figure 1.

Figure 1.

Intelligent evaluation process of Topic Talk items.

We input the speech of the candidate’s Topic Talk item into the decoder after preprocessing and feature extraction. Based on the acoustic model and the language model, the decoder can obtain an optimal output sequence by using Viterbi algorithm. The output of the recognizer is used as the reference text for the subsequent text-independent evaluation tasks. First, the posterior probability of the obtained reference text relative to the pronunciation quality is calculated, and then the features of various evaluation indicators such as pronunciation standard degree, fluency degree, and vocabulary and grammar standard are calculated based on the posterior probability. Finally the total machine score is predicted based on all the obtained feature values.

For text-independent pronunciation quality evaluation, it mainly depends on the posterior probability of the recognition result relative to the pronunciation quality. Therefore, improving the recognition rate of the front-end recognizer is especially important for the posterior probability estimation of the Topic Talk item. The acoustic model and the language model has a great impact on the recognition performance of the front-end recognizer. Since the acoustic model and the language model have a great impact on the recognition performance of the front-end recognizer, in this paper, we mainly optimize the LVSCR module from the two aspects of the acoustic model and the language model to improve the speech recognition rate of the front-end, so as to calculate the posterior probability more accurately and better measure the pronunciation quality of candidates.

3.

IMPROVED DBLSTM-HMM ACOUSTIC MODEL

Since Hidden Markov Model (HMM) can describe the relationship between the hidden state and feature sequence in speech information, it is widely used in acoustic modeling in speech recognition tasks. The traditional Gaussian Mixture Model-Hidden Markov Model (GMM-HMM)⁵ has the advantages of fast training speed and easy transplantation, but it lacks the learning ability of deep nonlinear feature transformation. The leading Deep Neural Network-Hidden Markov Model (DNN-HMM)⁶ can make better use of the relevant information between speech frames and has the ability to learn deep nonlinear feature transformations, but it lacks the ability to model the long-term correlation of speech. The Recurrent Neural Network-Hidden Markov Model (RNN-HMM)⁷ can better solve the modeling problem of long-term correlation information of speech, but as the number of network layers increases, problems such as gradient disappearance or explosion are prone to occur. The Long Short-Term Memory-Hidden Markov Model (LSTM-HMM)⁸ can effectively control the gradient disappearance or explosion problem in RNN, but like RNN, the memory of LSTM is unidirectional, when modeling the current moment, only historical information can be used, and future information cannot be introduced.

3.1

Network structure of DBLSTM

In order to realize the simultaneous modeling of front and rear bidirectional information, make up for the defect of unidirectional modeling in LSTM-HMM, and further improve the recognition rate, in this paper, we build a Deep Bidirectional Long Short-Term Memory model (DBLSTM) based on LSTM, as shown in Figure 2.

Figure 2.

Intelligent evaluation process of Topic Talk items.

We input the input sequence x into the forward LSTM layer and the backward LSTM layer at the same time. The forward hidden layer vector is iteratively calculated from front to back through the forward layer, and the backward hidden layer vector is iteratively calculated from back to front through the backward layer. We update the output sequence y = {y₁,y₂,….,y_M} based on two LSTM layers, where M is the number of output data. The iterative formula of the DBLSTM network is given as follows:

3.2

Improved DBLSTM acoustic model

With the deepening of Deep Neural Network, the gradient disappearance of DBLSTM will occur in both time domain and space domain⁹. We use the gate signal of linear cyclic connection in DBLSTM to deal with the gradient disappearance problem in the time domain. By introducing maxout neural network into DBLSTM to increase the depth of DBLSTM, the gradient disappearance problem in DBLSTM spatial domain can be solved by using maxout neuron to generate a constant gradient. In order to solve the problem of model over fitting in the process of DBLSTM training, we introduce dropout regularization algorithm. The improved DBLSTM depth mixing acoustic model proposed in this paper is shown in Figure 3.

Figure 3.

Depth mixed acoustic model of DBLSTM + Maxout.

The bottom layer of BLSTM in the figure mainly models the long-term related information of the input speech signal. The middle select connection layer transforms the output data of BLSTM network according to equation (3) and transmits it to the full connection layer. The maxout neurons in the full connection layer are regularized and trained according to the dropout algorithm, and finally enter the softmax output layer to output the results.

3.2.1

Maxout Neural Network.

The maxout neural network structure in the full connection layer is shown in Figure 4.

Figure 4.

Neural network structure of Maxout.

The activation unit group of Maxout neuron contains multiple optional activation units. We select the maximum value from the activation unit group as the output of Maxout according to equation (4):

where represents the output of the i-th Maxout neuron in the l-th layer, and k is the number of active units in the active unit group. represents the j-th activation unit of the i-th Maxout neuron in the l-th layer, which is obtained from the forward LSTM propagation layer of the previous layer:

where is the weight matrix from the neuron to the active unit z_l in the previous layer, and b_l is the bias vector. Maxout solves the gradient disappearance problem by generating a constant gradient during training. The gradient of maxout neuron is:

When the value of the active unit is the largest, the gradient of the Maxout neuron is 1, and the other is 0.

3.2.2

Dropout algorithm.

In order to avoid the over-fitting problem of DBLSTM network when there are few training samples, we introduce Dropout regularization algorithm. In the iterative process of the neural network, the weight update of the hidden layer nodes is set to prevent the over-fitting phenomenon of the network, that is, the weights of some hidden layer nodes are not updated in one iteration, but are activated and updated in the next iteration. Different regularization methods are used in the training and testing stages of the Dropout algorithm¹⁰.

• (1) Training stage

  We use a binary mask m_l in the original activation unit and calculate the output value of the Maxout neuron as follows:

  

  Among them, θ represents the nonlinear transformation operation of neurons, m_l obeys the 1 – r Bernoulli distribution, and r is the Dropout rate. The selection of the r value is very important in the network training process. The smaller the r value is, the more useful information is retained. The larger the r value, the higher the degree of regularization.

• (2) Testing stage

  In the test stage of dropout, it is not necessary to omit the activated neurons, but in order to compensate for Dropout training, it is necessary to reduce the activation value of neurons by 1 – r. In order to prevent Dropout regularization from damaging the long short term information of DBLSTM network¹¹, the Maxout network is only applied to the full connection layer.

4.

N-GRAM+RNN INTERPOLATION LANGUAGE MODEL

The intelligent evaluation of text-independent Topic Talk item depends greatly on the recognition rate of the front-end recognizer. If the decoding result of the front-end recognizer is wrong, the posterior probability calculated based on the recognition result will hardly provide useful information for the evaluation of pronunciation quality. A good language model can effectively improve the decoding efficiency of the recognizer, thereby improving the speech recognition rate.

The existing LVCSR recognizers generally use the n-gram language model based on statistics¹², which can only memorize the historical information of the previous two to three words, and the further historical information has no effect on the score of the current word. It is obviously that the lack of historical information reduces the reliability of n-gram language model scores. The language model based on the RNN¹³ can introduce further sentence history information in the training process, but the decoding efficiency of whose is not high. In order to improve the front-end speech recognition rate, we integrate the advantages of the two language models and perform interpolation operations on the n-gram language model and the RNN language model^{14, 15}. Firstly, we use the n-gram language model to obtain the one-pass decoding result of the decoder, and then use the RNN language model to re-estimate the score of the N-best candidate result¹⁶ decoded in one pass, and finally take the sentence with the highest re-estimated score as the new recognition result. The process is shown in Figure 5.

Figure 5.

Process of RNN rescoring identification result.

For the N-best candidate result decoded by the decoder in one pass with the n-gram language model, we keep its acoustic model score (AC Score) unchanged, and apply RNN to re-estimate the language model score (LM Score). Based on the original acoustic model score and the re-estimated language model score, we obtain a new score for each candidate sentence, and select the candidate sentence with the highest score as the new speech recognition result.

The language model score calculation formula for re-estimating the ith candidate sentence in the RNN language model is as follows:

Among them, AcScore_i is the acoustic model score of the i-th candidate sentence, which remains unchanged during the re-estimation of the language model score by using n-gram and RNN interpolation, W_i is the number of words in the i-th candidate sentence, C is the word penalty, λ is the interpolation coefficient, is the n-gram language model score, is the RNN language model score, and LmScale is the scaling factor of the language model score during decoding.

In the following text-dependent evaluation tasks, the reference text updated by the RNN can not only reduce the recognition errors caused by the language model, but also make the calculated posterior probability more conducive to the evaluation of pronunciation quality.

5.

POSTERIOR PROBABILITY ESTIMATION

After the one-pass decoding result of the candidate’s Topic Talk item is obtained through the front-end recognizer, the posterior probability of the decoded phoneme t is calculated as follows:

Among them, O = [o₁,o₂,…,o_N] is the acoustic observation vector corresponding to the decoded phoneme t. We assume that the prior probability p(q) of all phonemes is equal. In order to make the calculation of the posterior probability more targeted, the posterior probability denominator space Q_t is composed of the error prone phonemes of the decoded phoneme t [².

For the decoded phoneme t, if the optimal path obtained by Viterbi decoding is Θ = {s₁,s₂,…,s_N}, then ln p(O | t) can be approximately calculated as follows:

It is assumed that as long as the transition probability a_ij of the HMM is greater than zero, the jump from state i to state j can be completed, that is, the likelihood score calculation can ignore a_ij. For the DBLSTM-HMM acoustic model, p(o_i | s_i) can be calculated as follows:

Among them, p(s_i) is the prior probability of each HMM state obtained from the training set, p(o_i) can be regarded as a constant during the decoding process, and p(s_i | o_i) is the score of the neural network output corresponding to the state s_i after the softmax activation operation. The likelihood score of the DBLSTM-HMM acoustic model is given by using the following formula:

First, we estimate the posterior probability of each phoneme in the decoded phoneme according to equation (9), then calculate the average value of the posterior probability of all phonemes in a sentence, and finally calculate the average value of the posterior probability of all sentences in a speech, which is the final estimate of the posterior probability of the speech.

6.

EXPERIMENTAL VERIFICATIONS

7.

CONCLUSIONS

In order to realize the intelligent evaluation of Topic Talk item in PSC, we studied the text-independent oral evaluation task from the perspective of recognition, and constructed an intelligent evaluation model of Topic Talk item. Based on the recorded speech data of candidates collected from the PSC Centers across the country, we analyzed the speech recognition performance and word recognition performance of the proposed intelligent evaluation model from the perspectives of acoustic models and language models. Experiments show that the intelligent evaluation model proposed in this paper can better identify the text content of the candidate’s expression, and the posterior probability calculated based on the recognized text has a high correlation with the manual standard score. During the experiment, it was found that the contextual information of the recognition results was helpful for the improvement of the recognition performance, which also pointed out a direction for the follow-up research.