## Abstract

For most multistate Hopfield neural networks, the stability conditions in asynchronous mode are known, whereas those in synchronous mode are not. If they were to converge in synchronous mode, recall would be accelerated by parallel processing. Complex-valued Hopfield neural networks (CHNNs) with a projection rule do not converge in synchronous mode. In this work, we provide stability conditions for hyperbolic Hopfield neural networks (HHNNs) in synchronous mode instead of CHNNs. HHNNs provide better noise tolerance than CHNNs. In addition, the stability conditions are applied to the projection rule, and HHNNs with a projection rule converge in synchronous mode. By computer simulations, we find that the projection rule for HHNNs in synchronous mode maintains a high noise tolerance.

## 1  Introduction

Several multistate models of Hopfield neural networks have been studied in recent years. In particular, many researchers have studied complex-valued Hopfield neural networks (CHNNs) and have often applied them to the storage of image data (Aoki & Kosugi, 2000; Aoki, 2002; Hirose, 2003, 2012, 2013; Isokawa et al., 2018; Jankowski, Lozowski, & Zurada, 1996; Muezzinoglu, Guzelis, & Zurada, 2003; Nitta, 2009; Tanaka & Aihara, 2009; Zheng, 2014). CHNNs have been extended to the following hypercomplex-valued Hopfield neural networks:

1. Hyperbolic-valued Hopfield neural networks (HHNNs; Kuroe, Tanigawa, & Iima, 2011; Kobayashi, 2013, 2016, 2018c, 2019, 2020).

2. Quaternion-valued Hopfield neural networks (de Castro & Valle, 2017; Kobayashi, 2017a; Minemoto, Isokawa, Nishimura, & Matsui, 2016, 2017; Isokawa, Nishimura, Kamiura, & Matsui, 2006, 2007, 2008; Isokawa, Nishimura, & Matsui, 2012; Isokawa, Nishimura, Saitoh, Kamiura, & Matsui, 2008; Song & Chen, 2018; Valle, 2014; Valle & de Castro, 2016; Valle & de Castro, 2018).

3. Commutative quaternion-valued Hopfield neural networks (Isokawa, Nishimura, & Matsui, 2010; Kobayashi, 2018b).

Some of these models are implemented as alternatives to CHNN. HHNN provides the best noise tolerance (Kobayashi, 2018c). A rotor Hopfield neural network (RHNN) is another alternative to CHNN (Kitahara & Kobayashi, 2014). An RHNN is defined using vector-valued neurons and matrix-valued weights. It has double weight parameters of CHNN and HHNN and provides better noise tolerance than a CHNN or an HHNN. Under the usual conditions on the weights, a Hopfield neural network converges to a fixed point in asynchronous mode, whereas it converges or is trapped at a cycle of length 2 in synchronous mode (Kobayashi, 2017b). If it converges in synchronous mode, recall is expected to be much faster than in asynchronous mode. Unfortunately, a CHNN with a projection rule does not converge in synchronous mode. In fact, a CHNN was trapped at a cycle of length 2 by computer simulations (Kobayashi, 2017b). Although an RHNN converges in synchronous mode, it requires double-weight parameters (Kobayashi, 2018a). Thus, it is desirable that an HHNN converges in synchronous mode, because a CHNN and an HHNN have the same number of weight parameters. In this letter, we prove that an HHNN with a projection rule converges in synchronous mode. First, we provide the stability conditions of HHNNs in synchronous mode. Next, we prove that the projection rule satisfies the stability conditions. Thus, an HHNN with a projection rule converges in synchronous mode. We evaluate the HHNNs in synchronous mode by computer simulations, which support that the HHNNs in synchronous mode maintain the high noise tolerance of the asynchronous mode.

## 2  Hyperbolic Hopfield Neural Networks

A hyperbolic number is represented as $x+yu$ using real numbers $x$ and $y$. The imaginary unit $u$ satisfies $u2=+1$; that is, the imaginary unit of complex numbers is replaced with $u$. We should note that there exist zero divisors in hyperbolic numbers, unlike complex numbers. For example, the equality $(1+u)(1-u)=0$ holds. For the hyperbolic number $z=x+yu$, the real part $x$ is denoted as $Re(z)$. The norm of $z=x+yu$ is defined as
$|z|=x2+y2=Rez2.$
(2.1)
For the hyperbolic vector $z=(z1,z2,…,zn)T$, the norm is defined as
$|z|=∑i=1n|zi|2=RezTz.$
(2.2)
We present the HHNNs (Kobayashi, 2018c). Let $zi$ and $wij$ be the state of neuron $i$ and the weight from neuron $j$ to neuron $i$, respectively. We denote the weight matrix as $W$, whose $(i,j)$ element is $wij$. The weighted sum input to neuron $i$ is defined as
$Si=∑j=1Nwijzj,$
(2.3)
where $N$ is the number of neurons. The activation function for HHNNs is equivalent to the complex-valued multistate activation function. For the resolution factor $K$, we define $θK=πK$. For the weighted sum input $S=r(cosθ+usinθ)$, the activation function is defined as
$f(S)=10<θ<θKcos2θK+usin2θKθK<θ<3θK⋮⋮cos2(K-1)θK(2K-3)θK+usin2(K-1)θK<θ<(2K-1)θK1(2K-1)θK<θ<2π.$
(2.4)
We can describe the activation function as
$f(S)=argmaxv∈VRevS,$
(2.5)
where $V$ is the set of neuron states ${cos2kθK+usin2kθK}k=0K-1$. When the weighted sum input $S$ is on the decision boundary, there are multiple candidates of 2.5. Then the neuron state is supposed to remain. There are two update modes: asynchronous and synchronous. In asynchronous mode, multiple neurons are never updated at the same time. In synchronous mode, all the neurons are updated simultaneously. Suppose that neuron $i$ is updated from $zi$ to $zi'≠zi$. Since $zi'$ is the maximal argument of 2.5,
$Rezi'Si>ReziSi$
(2.6)
holds. Let $S$ be the weighted sum input vector $(S1,S2,…,SN)T$. In synchronous mode, suppose that the HHNN state is updated $z=(z1,z2,…,zN)T$ to $z'=(z1',z2',…,zN')T$, and there exists neuron $i$ such that $zi'≠zi$. Then, the inequality 2.6 implies
$Re(z')TS=∑i=1NRezi'Si$
(2.7)
$>∑i=1NReziSi$
(2.8)
$=RezTS.$
(2.9)
This inequality is necessary for the proof of theorem 2. In asynchronous mode, the following stability conditions are provided (Kobayashi, 2018c):
1. $WT=W$,

2. $diagW=O$.

When $W$ satisfies the stability conditions, the HHNN converges to a fixed point in asynchronous mode.

We describe the projection rule for HHNNs (Kobayashi, 2018c; Lee, 2006). Let $zp=(z1p,z2p,…,zNp)T$ be the $p$th training pattern. We denote the training matrix as $Z=(z1z2⋯zP)$, where $P$ is the number of training patterns. We define the weight matrix as
$W=Z(ZTZ)-1ZT.$
(2.10)
From $WZ=Z$, we have $Wzp=zp$, which implies that the training patterns are fixed points. The projection rule for HHNNs satisfies $WT=W$ but does not satisfy $diagW=O$. In the case of HHNNs, unlike that of CHNNs, the diagonal elements of $W$ cannot be eliminated. If the diagonal elements are eliminated, the training patterns are not stable, since the diagonal elements are not real numbers (Kobayashi, 2018c). Thus, the projection rule for HHNNs in asynchronous mode does not satisfy the stability conditions. However, in the computer simulations, all the trials converged to the fixed points. Since there must exist $(ZTZ)-1$, the storage capacity of the projection rule is $N$. The existence of the inverse matrices and the algorithm to determine them are described in the appendix.

## 3  Stability Conditions

We provide the stability conditions for HHNNs in synchronous mode. Since the elements of $W$ are hyperbolic numbers, we have to modify the definition of a nonnegative definite matrix to define the stability conditions for HHNNs in synchronous mode.

Definition 1.

Suppose that a hyperbolic matrix $M$ is symmetric. If $Re(zTMz)≥0$ for all $z$, then $M$ is said to be hyperbolic nonnegative definite.

We provide the stability conditions in synchronous mode:

1. $W$ is symmetric.

2. $W$ is hyperbolic nonnegative definite.

In the same way as with CHNNs and RHNNs, we can prove that if $W$ is symmetric, the HHNN converges to a fixed point or is trapped at a cycle of length 2 in synchronous mode (Kobayashi, 2018a). We prove that the HHNN converges to a fixed point, if the stability conditions are satisfied.

Theorem 1.

If $W$ satisfies the stability conditions, then the HHNN converges to a fixed point in synchronous mode.

Proof.
We define the energy as
$E=-12RezTWz.$
(3.1)
Suppose that the HHNN state is updated from $z$ to $z'≠z$. We calculate the energy change:
$ΔE=-12Rez'TWz'+12RezTWz$
(3.2)
$=-12Rez'-zTWz'-z-Rez'TWz+RezTWz.$
(3.3)
Since $W$ is hyperbolic nonnegative definite, we have
$Rez'-zTWz'-z≥0.$
(3.4)
From equation 2.9, we obtain
$Rez'TWz>RezTWz.$
(3.5)
Thus, the inequality $ΔE<0$ holds. Therefore, the HHNN converges to a fixed point. $□$

We apply theorem 2 to the projection rule for HHNNs; that is, we prove that an HHNN employing a projection rule converges to a fixed point in synchronous mode.

Theorem 2.

$W=ZZTZ-1ZT$ satisfies the stability conditions.

Proof.
First, we prove $W2=W$:
$W2=ZZTZ-1ZTZZTZ-1ZT$
(3.6)
$=ZZTZ-1ZTZZTZ-1ZT$
(3.7)
$=ZZTZ-1ZT$
(3.8)
$=W.$
(3.9)
Next, we prove that $W$ satisfies the stability conditions. We already know that $W$ is symmetric:
$RezTWz=RezTW2z$
(3.10)
$=RezTWTWz$
(3.11)
$=Re(Wz)T(Wz)$
(3.12)
$=|Wz|2≥0.$
(3.13)
Therefore, $W$ is hyperbolic nonnegative definite. $□$

CHNN and HHNN are particular instances of RHNN. It is known that an RHNN employing a projection rule converges in synchronous mode (Kobayashi, 2018a). An HHNN employing the projection rule converges in synchronous mode by theorems 2 and 3. However, a CHNN may be trapped at a cycle of length 2 (Kobayashi, 2017b).

## 4  Computer Simulations

We evaluate noise tolerance in synchronous mode by computer simulations and compare it with that in asynchronous mode. First, randomly generated training patterns and impulsive noise are employed. The parameters are $N=200$, $P=10,20,30$, and $K=32,64,128$. One hundred sets of training patterns are generated for a pair of $(K,P)$, and 100 trials are conducted for a training set; the total number of trials is 10,000. Impulsive noise is employed to decay a neuron state at a rate of $r$, where $r$ varies from 0.0 to 0.8 in steps of 0.05. The new state is randomly selected from $V$; the distribution of new states is uniform. If the original training pattern is retrieved, the trial is regarded as successful. Figure 1 shows the simulation result. The HHNNs in synchronous mode slightly outperform those in asynchronous mode. The difference increases as $P$ increases. The noise tolerance is almost independent of $K$ in both modes.

Figure 1:

Computer simulations employing the randomly generated training patterns and impulsive noise.

Figure 1:

Computer simulations employing the randomly generated training patterns and impulsive noise.

Next, the gray-scale images and gaussian noise are employed as examples of real data. The color images of the CIFAR-10 data set are transformed to 256-level gray-scale images of $32×32$ pixels for training data (Krizhevsky, 2009). Therefore, the resolution factor and number of neurons are fixed to $K=256$ and $N=1024$, respectively. Figure 2 shows some samples of gray-scale images. We attempt to retrieve the original images from those with gaussian noise. Figure 3 shows the samples of gray-scale images with gaussian noise. For a pair of $(P,σ)$, where $σ$ is the standard deviation, 1000 trials are conducted. The simulation result is shown in Figure 4. In both modes, the simulation results are almost identical. Figure 5 shows the average loop counts until convergence for $P=80$. The HHNNs in synchronous mode need twice as many loop counts as in asynchronous mode. All the neurons are simultaneously updated in synchronous mode, whereas neurons are not updated at the same time in asynchronous mode. Therefore, if the neurons of a synchronous HHNN are all updated in parallel, the asynchronous mode requires $N$ times the processing time of a synchronous loop. From the simulation results, asynchronous HHNN is about $N$/2 times slower than the synchronous mode

Figure 2:

These 256-level gray-scale images of $32×32$ pixels were employed as the training patterns.

Figure 2:

These 256-level gray-scale images of $32×32$ pixels were employed as the training patterns.

Figure 3:

These images are the training patterns with gaussian noise.

Figure 3:

These images are the training patterns with gaussian noise.

Figure 4:

Computer simulations employing the gray-scale images and gaussian noise.

Figure 4:

Computer simulations employing the gray-scale images and gaussian noise.

Figure 5:

Average loop counts until convergence.

Figure 5:

Average loop counts until convergence.

## 5  Conclusion

Many hypercomplex-valued Hopfield neural networks have been studied. Although stability in synchronous mode is necessary for parallel processing, stability conditions have never been provided for any hypercomplex-valued Hopfield neural network. In this study, we provide stability conditions for HHNN in synchronous mode. We also prove that the projection rule satisfies the stability conditions. Computer simulations show that the noise tolerance in synchronous mode maintains the high noise tolerance of HHNNs. In addition, we show that parallel processing accelerates recall. Although this theory has already been applied to RHNNs, it cannot be applied to CHNNs (Kobayashi, 2017b, 2018a). We plan to study the stability conditions for other hypercomplex-valued Hopfield neural networks.

## Appendix:  Algebra of Hyperbolic Numbers

The algebra of hyperbolic numbers is commutative. If the properties on real matrices do not need division, they are also true for hyperbolic matrices. For example, the following equalities are true:
$ABT=BTAT$
(A.1)
$A-1T=AT-1.$
(A.2)
For the hyperbolic square matrix $M$, $detM$ is defined in the same manner as the determinants of real matrices. Let $M˜$ be the adjugate matrix of $M$; then $MM˜=detMI$ holds. If $detM$ is not $r(1±u)$, where $r$ is a real number, there exist $detM-1$. Then, from $M((detM)-1M˜)=I$, $detM-1M˜$ is the inverse matrix of $M$. Gaussian elimination is available to determine the inverse of the hyperbolic matrix. We can also determine it by representing $x+yu$ as $xyyx$. For example, we determine $1+u1+2u1-uu-1$.
$1+u1+2u1-uu-1$
(A.3)
$⟶111211211-101-1110-1=1201-1-210-2-1-11111-111$
(A.4)
$⟶1+u1+2u1-uu-1=12u-1-2u-1+u1+u$
(A.5)

## References

Aoki
,
H.
(
2002
).
A complex-valued neuron to transform gray level images to phase information.
In
Proceedings of the International Conference on Neural Information Processing
,
3
(pp.
1084
1088
).
Piscataway, NJ
:
IEEE
.
Aoki
,
H.
, &
Kosugi
Y.
(
2000
).
An image storage system using complex-valued associative memories.
In
Proceedings of International Conference on Pattern Recognition
,
2
(pp.
626
629
).
Piscataway, NJ
:
IEEE
.
de Castro
,
F., Z.
, &
Valle
,
M., E.
(
2017
).
Continuous-valued quaternionic Hopfield neural network for image retrieval: A color space study.
In
Proceedings of Brazilian Conference on Intelligent Systems
(pp.
186
191
).
Washington, DC
:
IEEE Computer Society
.
Hirose
,
A.
(
2003
).
Complex-valued neural networks: Theories and applications.
Singapore
:
World Scientific
.
Hirose
,
A.
(
2012
).
Complex-valued neural networks
(2nd ed.).
Berlin
:
Springer
.
Hirose
,
A.
(
2013
).
Complex-valued neural networks: Advances and applications
.
Hoboken, NJ
:
Wiley
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2006
).
Fundamental properties of quaternionic Hopfield neural network.
In
Proceedings of International Joint Conference on Neural Networks
(pp.
610
615
).
Piscataway, NJ
:
IEEE
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2007
).
Dynamics of discrete-time quaternionic Hopfield neural networks.
In
Proceedings of International Conference on Artificial Neural Networks
(pp.
848
857
).
Berlin
:
Springer
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2008
).
Associative memory in quaternionic Hopfield neural network
.
International Journal of Neural Systems
,
18
,
135
145
.
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2010
).
Commutative quaternion and multi-state Hopfield neural networks.
In
Proceedings of IEEE World Congress on Computational Intelligence
(pp.
1281
1286
).
Piscataway, NJ
:
IEEE
.
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2012
).
On the fundamental properties of fully quaternionic Hopfield network.
In
Proceedings of International Joint Conference on Neural Networks
(pp.
1246
1249
).
Piscataway, NJ
:
IEEE
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Saitoh
,
A.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2008
).
On the scheme of quaternionic multistate Hopfield neural network.
In
Proceedings of Joint 4th International Conference on Soft Computing and Intelligent Systems and 9th International Symposium on Advanced Intelligent Systems
(pp.
809
813
).
Isokawa
,
T.
,
Yamamoto
,
H.
,
Nishimura
,
H.
,
Yumoto
,
T.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2018
).
Complex-valued associative memories with projection and iterative learning rules
.
Journal of Artificial Intelligence and Soft Computing Research
,
8
,
237
249
.
Jankowski
,
S.
,
Lozowski
,
A.
, &
,
J. M.
(
1996
).
Complex-valued multistate neural associative memory
.
IEEE Transactions on Neural Networks
,
7
,
1491
1496
.
Kitahara
,
M.
, &
Kobayashi
,
M.
(
2014
).
Projection rule for rotor Hopfield neural networks
.
IEEE Transactions on Neural Networks and Learning Systems
,
25
,
1298
1307
.
Kobayashi
,
M.
(
2013
).
Hyperbolic Hopfield neural networks
.
IEEE Transactions on Neural Networks and Learning Systems
,
24
,
335
341
.
Kobayashi
,
M.
(
2016
).
Global hyperbolic Hopfield neural networks
.
IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences
,
E99-A
,
2511
2516
.
Kobayashi
,
M.
(
2017a
).
Quaternionic Hopfield neural networks with twin-multistate activation function
.
Neurocomputing
,
267
,
304
310
.
Kobayashi
,
M.
(
2017b
).
Fast recall for complex-valued Hopfield neural networks with projection rules
.
Intelligence and Neuroscience
,
2017
,
4894278
.
Kobayashi
,
M.
(
2018a
).
Stability of rotor Hopfield neural networks with synchronous mode
.
IEEE Transactions on Neural Networks and Learning Systems
,
29
,
744
748
.
Kobayashi
,
M.
(
2018b
).
Twin-multistate commutative quaternion Hopfield neural networks
.
Neurocomputing
,
320
,
150
156
.
Kobayashi
,
M.
(
2018c
).
Hyperbolic Hopfield neural networks with directional multistate activation function
.
Neurocomputing
,
275
,
2217
2226
.
Kobayashi
,
M.
(
2019
).
Storage capacity of hyperbolic Hopfield neural networks
,
Neurocomputing
,
369
,
185
190
.
Kobayashi
,
M.
(
2020
).
Noise robust projection rule for hyperbolic Hopfield neural networks
.
IEEE Transactions on Neural Networks and Learning Systems
,
31
,
352
356
.
Krizhevsky
,
A.
(
2009
).
Learning multiple layers of features from tiny images
(Technical Report)
.
Toronto University of Toronto
.
Kuroe
,
Y.
,
Tanigawa
,
S.
, &
Iima
,
H.
(
2011
).
Models of Hopfield-type Clifford neural networks and their energy functions: Hyperbolic and dual valued networks
. In
Proceedings of International Conference on Neural Information Processing
(pp.
560
569
).
Berlin
:
Springer
.
Lee
,
D. L.
(
2006
).
Improvements of complex-valued Hopfield associative memory by using generalized projection rules
,
IEEE Transactions on Neural Networks
,
17
,
1341
1347
.
Minemoto
,
T.
,
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2016
).
Quaternionic multistate Hopfield neural network with extended projection rule.
Artificial Life and Robotics
,
21
(
2016
)
106
111
.
Minemoto
,
T.
,
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2017
).
Pseudo- orthogonalization of memory patterns for complex-valued and quaternionic associative memories
.
Journal of Artificial Intelligence and Soft Computing Research
,
7
,
257
264
.
Muezzinoglu
,
M. K.
,
Guzelis
,
C.
, &
,
J. M.
(
2003
).
A new design method for the complex-valued multistate Hopfield associative memory
.
IEEE Transactions on Neural Networks
,
14
,
891
899
.
Nitta
,
T.
(
2009
).
Complex-valued neural networks: Utilizing high-dimensional parameters
.
Hershey, PA
:
Information Science Publishing
.
Song
,
Q.
, &
Chen
,
X.
(
2018
).
Multistability analysis of quaternion-valued neural networks with time delays
.
IEEE Transactions on Neural Networks and Learning Systems
,
29
,
5430
5440
.
Tanaka
,
G.
, &
Aihara
,
K.
(
2009
).
Complex-valued multistate associative memory with nonlinear multilevel functions for gray-level image reconstruction
.
IEEE Transactions on Neural Networks
,
20
,
1463
1473
.
Valle
,
M. E.
(
2014
).
A novel continuous-valued quaternionic Hopfield neural network.
In
Proceedings of Brazilian Conference on Intelligent Systems
(pp.
97
102
).
Piscataway, NJ
:
IEEE
.
Valle
,
M., E.
, &
de Castro
,
F., Z.
(
2016
).
Theoretical and computational aspects of quaternionic multivalued Hopfield neural networks.
In
Proceedings of International Joint Conference on Neural Networks
(pp.
4418
4425
).
Piscataway, NJ
:
IEEE
.
Valle
,
M., E.
, &
de Castro
,
F., Z.
, (
2018
).
On the dynamics of Hopfield neural networks on unit quaternions
.
IEEE Transactions on Neural Networks and Learning Systems
,
29
,
2464
2471
.
Zheng
,
P.
(
2014
).
Threshold complex-valued neural associative memory.
IEEE Transactions on Neural Networks and Learning Systems
,
25
,
1714
1718
.