1.

INTRODUCTION

In recent years, researches in minimization have rising heat, due to its essential position in artificial intelligence fields, such as the control of mechanical arms ¹, health big data digitization ² and logistics engineering ³. Many researchers have developed numerous algorithms based on traditional machine learning techniques and the latest deep learning techniques. Although there are many minimization algorithms, a considerable part of them are designed to solve time-invariant problems. But in reality there are still time-varying problems that need to be addressed, and with multiple time-varying applications in both industry and research fields, it is of great importance to develop minimization model to deal with time-varying problems.

Among many minimization algorithms, ZNN is widely used due to its special structure. At the same time, due to the particularity of its own structure, there are natural similarities between the types of problems it can deal with and time-varying problems, so it has been receiving extensive attention. Specifically, zeroing neural network (ZNN), a specific kind of recurrent neural network (RNN), is a product of Hopfield neural network research. Recurrent neural network is a kind of neural network with a special structure, and its main processing core is a computing unit of cyclic operation, which can realize parameter sharing and state storage. It is capable of receiving long sequences of information, such as a piece of text or a piece of speech. After receiving the sequence information, the cyclic neural network can perform operations in sequence, and at the same time, it can save the above information to an intermediate state, and comprehensively consider when processing the following information to complete context awareness. Zeroing neural network is a special case of neural network with this structure, which has the structural characteristics of RNN and also has some independent designs. For its parallel distributed design and practical hardware implementation, ZNN is suggested as a methodical strategy to address time-varying problems. For instance, in ⁴, to solve the Lyapunov equation, researchers suggest using a convergence-accelerated zeroing neural network, which is frequently seen in the sciences and engineering. And researchers in ⁵ discover that when used to control a nonlinear dynamical system, ZNN outperforms controller based on linear quadratic regulator (LQR).

These studies have fully proved that the neural network processing of zeroing neural network (ZNN) can handle time-varying problems well. Hence, zeroing neural network (ZNN) is created to address a variety of time-varying issues in practical engineering applications ⁶. The iteration of a neural network is discrete in all real-world engineering issues. This characteristic of ZNN enables it to handle some engineering problems involving time-varying, but there is another point that cannot be ignored in the application process, which is the processing of nonlinear time-varying, that is, discrete time-varying problems. To make ZNN play a role in practical engineering problems and reduce time-varying nonlinearity, discrete-time ZNN (DTZNN), which originates from discretization of continuous-time zeroing neural network (CTZNN), can be thought of as a useful tool for resolving discrete time-varying issues ⁷.

After paying attention to the need to solve the discretization problem, researchers have proposed many types of algorithms, including further deformation of DTZNN and derivation algorithm from CTZNN to DTZNN. During the process of DTZNN building, it is necessary to discretize the CTZNN using the proper formula for discretization. Up to now, an Euler-type DTZNN using the Euler forward-difference formula has been presented ⁸. In order to carry out the discrete-time varying nonlinear minimization, researchers have also developed Taylor-type DTZNN based on Taylor-type differential method ^9-10. Additionally, a general three-step technique is created by the researchers to discretize the CTZNN into a DTZNN ¹¹.

All the above-mentioned discretization methods for CTZNN have achieved good results in practice, but there is a key problem, that is, the operational flexibility in the discretization process is limited. Intuitively speaking, an adjustable discretization process will also have a great impact on the final effect. In this paper, to further optimize and add more flexibility to the related DTZNN, we present the general four-instant discretization formula ¹², which has an adjustable parameter d, for the discretization process. These parameters on the one hand represent the flexibility of the discrete process, and on the other hand have an impact on the final result. On this basis, we conducted several numerical verification experiments. First, we designed control variable experiments to explore the flexibility of the method. And we designed experiments to explore the trend of the residual as d changes, in order to confirm the impact of the method on the final effect. The results show that with the change of d, the convergence curve of the residual error also changes. For the design of ZNN, the new four-instant discretization formula offers greater operability, so that the new ZNN theory can deal with more kinds of engineering problems.

2.

PROBLEM FORMULATION

Before discretization, the CTZNN approach can be used to create the CTZNN model initially. The following time-varying minimization issue ⁹ is initially thought of as

where the second-order differentiable and bounded f (·,·): ℝⁿ ×[0, +∞) → ℝ is a time-varying nonlinear mapping function. In ZNN theory, f (x(t),t) is also considered as a time-varying signal, while x(t) is always considered as neural state.

Calculating the gradient of f (x(t),t) is an important step to address the minimization problem in CTZNN. By using the matrix differentiation rule, gradient ψ(x(t),t) is derived:

which derives from f (x(t), t) and is an array of nonlinear differentiable mapping functions. It can be concluded from the idea of derivative that, to solve the minimization problem, ψ(x(t), t) is required to be zero.

After that, we use the ZNN design formula, e(t) = de(t) / dt = –γe(t) ¹³, where the vector-valued error function is defined as e(t) = ψ (x(t), t), and design parameter γ > 0 is used to adjust the convergence rate of ZNN. The dynamical formula of CTZNN below is obtained by expanding the ZNN design formula:

In equation, H (x(t), t) and are defined as

and

Thus, is translated as follow for the nonsingular condition of H(x(t), t) that we investigate in the research:

After discreting continous time-varying signal in equation on the time interval [t₀, t_f], the equation that follows can be formed as

Since it is a given that sampling at time instant t = (k + 1)δ (abbreviated as t_{k +1}) creates and measures the signal f (x_{k +1}, t_{k +1}) from the signal f (x(t), t) with a smooth time variation. Additionally, the update index is indicated by k = 0,1, 2⋯, while the sampling gap is indicated by δ > 0.

To solve the discrete-time nonlinear minimization shown in equation, equation is discretized to generate following equation as

In order to solve the time-varying minimization issue using DTZNN, x_{k +1} need to be generated between t_k and t_{k +1}. To realize this, proper theory for discretization and differentiation is required to ensure the computing efficiency, while flexibility should also be considered. In this paper, for the first time, we use the four-instant discretization formula in the differential step ¹², which is shown in following Lemma 1.

Lemma 1 There is an O truncation Error O(δ²) in the second-order derivative elimination (SODE) method’s four-instant discretization formula. The effective domain of parameter d is d ∈ (–∞, –1) ∪ (1 / 3, +∞) for the ZNN discretized by such a formula.

The general four-instant discretization formula is given as follow:

After that, the Taylor-type DTZNN as follow can be created by discretizing CTZNN in equation using the aforementioned equation

where d is a parameter that can be adjusted within (–∞, –1) ∪ (1 / 3, +∞). When d is set to be 1, the following equation of x_{k +1} is derived as

which is the exact outcome of utilizing Taylor-type differentiation formula for discretization ⁸. From this result, it can be deduced that for the four-instant general discretization formula, the Taylor-type differentiation formula is a specific instance. So compared with Taylor-type differentiation formula, the four-instant general discretization formula gives the novel DTZNN more tunability and flexibility in various engineering problems.

3.

NUMERICAL EXPERIMENTS AND VERIFICATIONS

In our proposed method, the main focus is on the flexibility embodied in the discretization process, which affects the generalization of the algorithm and the final experimental effect. In order to further verify the actual effect of the algorithm, we designed two parts of experiments. The first part of the experiment focuses on the relationship between the selection of different values and the order of the residuals. Specifically, we verify the flexibility of the algorithm by observing the order of the residuals under different values. In the second part, we mainly focus on the impact of the value on the final effect, and explore an appropriate value strategy.

We present a number of numerical experiments in order to further investigate numerous properties of the novel DTZNN. During the experiments, we used the control variate method by selecting different values of δ and d. To some extent, the experimental findings reflect the features and advantages of the new DTZNN.

Take into account the subsequent discrete time-varying minimization problem, where each computation time period [kδ, (k + 1)δ] ⊆ [0,10] should compute x_{k +1} with update index k = 0,1, 2⋯:

in which x_k = [x₁(t_k), x₂(t_k), x₃(t_k, x₄(t_k)]^T.

In the following two experiments, we created two time-varying mapping functions as examples to show more general experimental findings as

and

3.1

First Experiment

The first experiment is designed to analyze the evolution of residual error. In this experiment, we study the change of error with the change of δ and prove the order of residual error to be O(δ³).

In DTZNN, h = γδ > 0 is often used to represent the iteration step size. In this experiment, to control the variable, d is set to be constant 1 and h is set to be constant 0.3. The initial vector x₀ ∈[–0.25, 0.25]⁴ is a vector which is generated randomly. Besides, the following two vectors x₁ and x₂ are generated by x₀ according to the Newton iteration, for its structure is the most basic. The formula of Newton iteration is shown in equation :

After that, x₄ and following neural states are generated according to equation. To study the change of error with the change of δ, the value of δ is set to be 0.01 and 0.001 respectively, and applied in the iteration process of f₁ (x_k, t_k) and f₂ (x_k, t_k) for comparison. In Fig. 1, the residual error is displayed, in which t = (k + 1)δ. ∥R₁ (t)∥ and ∥R₂ (t)∥ are defined as

Fig. 1:

Semilog plot of residual errors of two time-varying signal f₁ (x_k, t_k) and f₂ (x_k, t_k), where t = (k + 1)δ and h = 0.3, using formula shown in equation.

and

Fig. 1 (a) shows the semilog plot of the errors of f₁ (x_k, t_k) when δ changes. Additionally, Fig1. (b) displays a semilog plot showing the error of f₂ (x_k, t_k) when δ changes. From Fig. 1, it can be observed that with t increasing, residual error decreases faster when δ = 0.001 than when δ = 0.01. From equation, it is clear that the order of the residual error is O(δ³). Besides, from Fig. 1, it is apparent that the residual error associated with δ = 0.001 is 1 / 1000 of the error associated with δ = 0.01 when the curve tends to be flat and t is established. Therefore, the order of residual error in equation can be proved.

3.2

Second Experiment

The second experiment is aimed to study the variation of residual error with d. Same as the first experiment, a vector named x₀ ∈[–0.25, 0.25]⁴ is produced at random. In addition, x₀ generates the other two vectors, x₁ and x₂, in accordance with the Newton iteration. After that, equation produces x₄ and the subsequent neural states.

In contrast to the first experiment, to control variables, δ is fixed to be constant at 0.001 and h is set to be constant at 0.3 in the second experiment. We no longer take t as the horizontal axis, but k, which is the update index. From Lemma 1, the area in which parameter d is effective is d ∈ (–∞, –1) ∪ (1 / 3, +∞). Therefore, to ensure the universality of results, we take -4, -2, 1, 2 and 4 respectively as the different values of d. By applying different values of d, different residual error curves are got, shown in Fig. 2.

Fig. 2:

Semilog plot of residual errors of two time-varying signal f₁ (x_k, t_k) and f₂ (x_k, t_k), where δ = 0.001 and h = 0.3, using formula shown in equation.

The errors of f₁ (x_k, t_k) when d varies is shown in a semilog plot Fig. 2 (a), and the semilog plot of the errors of f₂ (x_k, t_k) when d changes is shown in Fig. 2 (b). In order to better observe the law that error changes with d, the curve corresponding to d = 1 can be regarded as a benchmark curve, because when d = 1, the trend of error change is the result of iteration using Taylor formula, shown in equation.

By adjusting the value of d while ensuring that other parameter values remain unchanged, the experimental results show some characteristics of novel DTZNN as follows.

• • When d is not taken as 1, the error decreases faster than when d is taken as 1.

• • When d is negative, error decreases faster than when d is positive.

• • When is positive, the error drops more quickly the higher the absolute value of d.

• • When is negative, the error drops more quickly the smaller the absolute value of d.

The above four properties demonstrate the effectiveness of our proposed method, and the choice of d value will indeed have a significant impact on the final effect. Users can choose d in practical application according to the characteristics of the algorithm, which takes into account the flexibility and effectiveness of the algorithm. For example, according to the experimental results, it is preferred to set d as a negative number and the absolute value is as small as possible during the use of the algorithm.

From the results shown in first experiment and second experiment, it is clear that the order of residual error is O(δ³). In addition, the experiment also verified the flexibility of our proposed method. The main process is to specialize the value of d to obtain a fixed Taylor-type DTZNN. Experiments show that as a special case of the novel DTZNN, the Taylor-type DTZNN (in which d = 1) shown in equation performs worse than other cases. Such experimental results verify the effectiveness of our idea of focusing on flexibility in the discrete process, and the adjustment of the d value also provides a comparison standard. Therefore, the result shows that the new DTZNN has effectiveness dealing with discrete time-varying minimization problem.

4.

CONCLUSION

To solve a range of time-varying difficulties in real-world engineering applications, the zeroing neural network (ZNN) was developed. However, these methods do not pay much attention to the flexibility of the discrete process, which will affect the final effect of the model. In order to solve discrete time-varying minimization problems, we have developed a novel discrete-time ZNN (DTZNN) in this paper. To get the novel DTZNN, continuous time-varying minimization problem has been firstly considered. After applying ZNN design formula, we use a different way to discretize CTZNN. For the first time, four-instant discretization formula with an adjustable parameter d has been used. To verify the performance of novel DTZNN, two time-varying mapping functions have been designed as examples. What’s more, two numerical experiments are designed to study the properties of the model. In the first experiment, different values of δ is chosen to observe the order of residual error. In the second numerical experiments, δ is fixed to be constant at 0.001, and different values of d have been selected respectively to observe its impact on the residual error. The outcomes of numerical experiments prove the order of residual error to be O(δ³) in novel DTZNN, and the flexibility of the new DTZNN. To sum up, the numerical experiments demonstrate the usefulness of novel DTZNN in solving discrete time-varying minimization problem.