Abstract

A complex-valued Hopfield neural network (CHNN) with a multistate activation function is a multistate model of neural associative memory. The weight parameters need a lot of memory resources. Twin-multistate activation functions were introduced to quaternion- and bicomplex-valued Hopfield neural networks. Since their architectures are much more complicated than that of CHNN, the architecture should be simplified. In this work, the number of weight parameters is reduced by bicomplex projection rule for CHNNs, which is given by the decomposition of bicomplex-valued Hopfield neural networks. Computer simulations support that the noise tolerance of CHNN with a bicomplex projection rule is equal to or even better than that of quaternion- and bicomplex-valued Hopfield neural networks. By computer simulations, we find that the projection rule for hyperbolic-valued Hopfield neural networks in synchronous mode maintains a high noise tolerance.

1  Introduction

A complex-valued Hopfield neural network (CHNN) is one of the first multistate models of Hopfield neural networks and has been studied by many researchers (Aizenberg, Ivaskiv, Yu, Pospelov, & Hudiakov, 1971, 1973; Jankowski, Lozowski, & Zurada, 1996; Noest, 1988). The CHNN has been used as neural associative memories and applied to storage of multilevel image data (Aoki & Kosugi, 2000; Aoki, 2002; Lee, 2006; Muezzinoglu, Guzelis, & Zurada, 2003; Tanaka & Aihara, 2009; Zheng, 2014). It has also been extended using hypercomplex numbers (Hitzer, Nitta, & Kuroe, 2013). For example, several models of quaternion-valued Hopfield neural networks (QHNNs) have been proposed. Isokawa et al. (Isokawa, Nishimura, Kamiura, & Matsui, 2006, 2007, 2008) proposed a QHNN with a split activation function. The activation function that was extended to a phasor-represented activation function (Isokawa, Nishimura, Saitoh, Kamiura, & Matsui, 2008; Isokawa, Nishimura, & Matsui, 2009; Minemoto, Isokawa, Nishimura, & Matsui, 2016). A continuous activation function for QHNNs has also been proposed (Valle & de Castro, 2018; de Castro & Valle, 2017; Valle, 2014). Bicomplex numbers, also referred to as commutative quaternions, are hypercomplex numbers of dimension 4, like quaternions. Isokawa, Nishimura, and Matsui (2010) proposed a bicomplex-valued Hopfield neural network (BHNN) with a phasor-represented activation function.

Several models have been proposed as alternatives of CHNN. The models with more weight parameters can provide better noise tolerance. Since the weight parameters need a lot of resources, the number of these parameters should be reduced. Twin-multistate activation functions were introduced to the QHNNs and BHNNs for reduction in the number of weight parameters (Kobayashi, 2017, 2018a). However, the architectures of a QHNN and a BHNN with twin-multistate activation functions are more complicated than that of a CHNN. Since a CHNN has simple architecture and has been the most widely used multistate Hopfield model, reducing the weight parameters of a CHNN is desirable. In this work, the number of weight parameters of a CHNN is reduced by the bicomplex projection rule. A projection rule is a one-shot learning algorithm, and the learning speed is fast, unlike iterative learning algorithm, such as gradient descent learning and relaxation algorithm. Gradient descent learning was proposed by Lee (2001) and was improved by Kobayashi et al. (Kobayashi, 2016; Kobayashi, Yamada, & Kitahara, 2011). The relaxation learning was proposed by Muezzinoglu et al. (2003) and was reformulated by Kobayashi (2008). Although the Hebbian rule is a one-shot learning algorithm, the storage capacity is too small (Jankowski et al., 1996). A CHNN with a bicomplex projection rule has only half the weight parameters of a conventional CHNN, like a QHNN and a BHNN with twin-multistate activation functions. The proposed CHNN has much simpler architecture than the QHNN and BHNN. A bicomplex projection rule is obtained by decomposition of a BHNN with a projection rule. In addition, the proposed CHNN improves the noise tolerance of BHNN by removing self-feedbacks. BHNN's feedback is a major factor of reduction in noise tolerance. To evaluate the bicomplex projection rule, we compare the proposed CHNN with the QHNN and BHNN by computer simulations. They have half the weight parameters of a conventional CHNN.

2  Complex-Valued Hopfield Neural Networks

We briefly describe the CHNNs with N neurons (Jankowski et al., 1996). Let za and wab be the state of neuron a and the weight from neuron b to neuron a, respectively. The weights satisfy the stability conditions
wab=wba¯,
(2.1)
waa=0.
(2.2)
Let N be the number of neurons. The weighted sum input to neuron a is
Sa=b=1Nwabzb.
(2.3)
For resolution factor K, we denote θK=2π/K. The set of neuron states is defined as
H=hl=explθKil=0K-1.
(2.4)
The activation function is
f(za,Sa)=argmaxhHReh¯Sa.
(2.5)
When the right side of equation 2.5 has the multiple maximum arguments, the value of activation function cannot be determined. Then we define f(za,Sa)=za.

3  Bicomplex-Valued Hopfield Neural Networks

Bicomplex numbers form a commutative ring of dimension 4 and have the unipotent element j such that j2=+1 (Catoni, Cannata, & Zampetti, 2006). A bicomplex number is represented by γ=α+βj using complex numbers α and β. For γ=α+βj and γ'=α'+β'j, the addition and multiplication are defined as
γ+γ'=(α+α')+(β+β')j,
(3.1)
γγ'=(αα'+ββ')+(αβ'+α'β)j,
(3.2)
respectively. The conjugate is defined as
γ¯=α¯+β¯j.
(3.3)
See appendix for implementation of bicomplex numbers.
A neuron of BHNN has two multistate components, that is, a pair of complex-valued neurons. Suppose that N is even. We define bicomplex-valued neuron a as ya=z2a-1+z2aj. Since a bicomplex-valued neuron consists of two complex-valued neurons, a CHNN is transformed to a BHNN with N/2 neurons. The weight from bicomplex-valued neuron b to bicomplex-valued neuron a is denoted as vab=sab+tabj. The weight matrix is denoted as V, whose (a,b) element is vab. The weights satisfy the stability conditions (de Castro & Valle, 2020)
vab=vba¯,
(3.4)
vaa0.
(3.5)
For the weighted sum input
Sa=Aa+Baj=b=1N/2vabyb,
(3.6)
the activation function for BHNNs is
g(ya,Sa)=f(z2a-1,Aa)+f(z2a,Ba)j,
(3.7)
that is, the complex-valued activation function is independently applied to two components.
The projection rule for BHNNs is briefly described in Kobayashi (2018a). Let yp=(y1p,y2p,,yN/2p)T(p=1,2,,P) be the pth training pattern, where P is the number of training patterns. The training matrix is defined as
Y=y11y12y1Py21y22y2PyN/21yN/22yN/2P.
(3.8)
The projection rule provides the weight matrix by
V=Y(Y*Y)-1Y*.
(3.9)
Y* is the Hermitian conjugate of Y. The projection rule for BHNNs does not satisfy the stability condition, equation 3.5. There are the self-feedbacks that deteriorate the noise tolerance. Since the self-feedbacks are not real numbers, if they are removed, the training patterns are not fixed points. In fact, VY=Y implies
yap=b=1N/2vabybp.
(3.10)
If the self-feedback vaa is removed, the weighted sum input is (1-vaa)yap. To maintain the phase of yap, it is necessary that vaa is a real number. In general, vaa is not a real number. This problem will be solved by the decomposition of BHNN, referred to as the bicomplex projection rule for CHNNs, as described in section 4.

4  Bicomplex Projection Rule for Complex-Valued Hopfield Neural Networks

In this work, the bicomplex projection rule for CHNNs is provided by the decomposition of BHNNs. We show that an BHNN with N/2 neurons can be regarded as a CHNN with N neurons. Suppose ab. In Figure 1, the upper panel illustrates the weight from bicomplex-valued neuron b to bicomplex-valued neuron, a. A connection between bicomplex-valued neurons is decomposed to that between complex-valued neurons as illustrated by the lower panel. From vabyb=sabz2b-1+tabz2b+(tabz2b-1+sabz2b)j, we obtain
w2a-12b-1=w2a2b=sab,
(4.1)
w2a2b-1=w2a-12b=tab.
(4.2)
From vab¯=vba, we obtain sab¯=sba and tab¯=tba. Thus, stability condition 2.1 for CHNNs is satisfied. Since two connections share a weight parameter, the number of weight parameters is half that of conventional CHNN. Next, the self-feedbacks of BHNN are decomposed, as illustrated in Figure 2. We note that saa and taa are real numbers from V*=V. In the decomposition, the self-feedback saa is a real number. The bicomplex projection rule provides
w2a-12a-1=w2a2a=0,
(4.3)
w2a-12a=w2a2a-1=taa.
(4.4)
Thus, the self-feedbacks of CHNN are removed and the stability condition 2.2 is satisfied.
Figure 1:

Decomposition of connection between bicomplex-valued neurons.

Figure 1:

Decomposition of connection between bicomplex-valued neurons.

Figure 2:

Decomposition of self-feedback of bicomplex-valued neuron.

Figure 2:

Decomposition of self-feedback of bicomplex-valued neuron.

5  Computer Simulations

We evaluate the noise tolerance of CHNN with the bicomplex projection rule by computer simulations. Since the comparison should be conducted under the fixed number of weight parameters, it is compared with a QHNN and a BHNN with twin-multistate activation functions. They have half-weight parameters of a conventional CHNN. The results of a conventional CHNN are provided only as a reference and are not discussed. First, randomly generated training patterns and impulsive noise are employed. The parameters P and K vary, and N=200 is fixed. We generate 100 training pattern sets and conduct 100 trials for each. Impulsive noise is added by replacing each neuron component with a new randomly selected state at the rate r. After a training pattern is randomly selected, the noise is added. If the original pattern is retrieved, the trial is regarded as successful. Figure 3 shows the simulation results. The BHNN underperforms the QHNN and proposed CHNN, and the results of QHNN and proposed CHNN are identical. The noise tolerance of BHNN rapidly deteriorates with an increase in P.
Figure 3:

Computer simulations using random data.

Figure 3:

Computer simulations using random data.

Next, real image data and gaussian noise are employed. The CIFAR-10 data set (Krizhevsky, 2009) is transformed to the training data set, which consists of 256-level gray-scale images of 32×32 pixels. Therefore, K=256 and N=1024 are fixed. Figure 4 provides training samples. The standard deviation σ ranges from 0 to 30 in steps of 2. Figure 5 shows some images with gaussian noise. The simulation results are shown in Figure 6. The BHNN underperforms the QHNN and proposed CHNN. The noise tolerance of BHNN rapidly deteriorates with an increase in P again. For P=20, the BHNN and QHNN show similar results, otherwise the BHNN slightly outperforms the QHNN.
Figure 4:

Training data.

Figure 4:

Training data.

Figure 5:

Training data with gaussian noise.

Figure 5:

Training data with gaussian noise.

Figure 6:

Computer simulations using image data.

Figure 6:

Computer simulations using image data.

In both simulations, only the noise tolerance of BHNN rapidly deteriorates with an increase in P. Although the CHNN with the bicomplex projection rule is just a decomposition of BHNN, it is more robust against an increase in P. In other words, the decomposition of BHNN improves the noise tolerance. The chief difference is whether there are self-feedbacks. In many different models, such as hyperbolic and rotor Hopfield neural networks, the similar results have been pointed out and discussed (Kitahara & Kobayashi, 2014; Kobayashi, 2018b, 2020).

6  Conclusion

To reduce the number of weight parameters, twin-multistate activation functions have been introduced to QHNNs and BHNNs. Since their architectures are complicated, alternatives with simple architectures are necessary. In this work, the bicomplex projection rule is introduced to the CHNNs, where two connections share a weight parameter by decomposition of BHNN. This method can remove self-feedback and improves the noise tolerance of BHNN. Computer simulations support that the noise tolerance of proposed CHNNs is better than or equals that of QHNNs. Table 1 summarizes the self-feedbacks and update counts of a loop. The noise tolerance of BHNNs deteriorates by the self-feedbacks. Since a QHNN and a BHNN can update two multistate components simultaneously, their update counts of a loop are that of CHNN.

Table 1:
Summary of Properties.
ModelSelf-FeedbacksUpdate Count
QHNN Do not exist N/2 
BHNN Exist N/2 
Proposed Do not exist N 
ModelSelf-FeedbacksUpdate Count
QHNN Do not exist N/2 
BHNN Exist N/2 
Proposed Do not exist N 

Appendix

For readers who are not familiar with operation of bicomplex numbers, matrix representation is introduced. A bicomplex number γ=α+βj is transformed to T(γ)=αββα. For γ=α+βj and γ'=α'+β'j,
T(γ)T(γ')=αββαα'β'β'α'
(A.1)
=αα'+ββ'αβ'+βα'αβ'+βα'αα'+ββ'
(A.2)
=T(γγ')
(A.3)
holds. The inverse matrix can be determined using matrix representation. We determine the inverse matrix of X=(1+i)+j1-j1+j-i+(1+i)j as an example:
T(X)=1+i11-111+i-1111-i1+i111+i-i,
(A.4)
T(X)-1=151-3i1+2i-2-i2+i1+2i1-3i2+i-2-i-2+i-2+i2+i3-i-2+i-2+i3-i2+i,
(A.5)
X-1=15(1-3i)+(1+2i)j(-2-i)+(2+i)j(-2+i)+(-2+i)j(2+i)+(3-i)j.
(A.6)
By matrix representation, we can implement the BHNNs without bicomplex numbers. However, the disadvantage is that more memories are needed.

References

Aizenberg
,
N. N.
,
Ivaskiv
,
Yu, L.
,
Pospelov
,
D. A.
, &
Hudiakov
,
G. F.
(
1971
).
Multivalued threshold functions. I. Boolean complex-threshold functions and their generalization
.
Cybernetics and Systems Analysis
,
7
,
626
635
.
Aizenberg
,
N. N.
,
Ivaskiv
,
Yu
, L.,
Pospelov
,
D. A.
, &
Hudiakov
,
G. F.
(
1973
).
Multivalued threshold functions. II. Synthesis of multivalued threshold elements
.
Cybernetics and Systems Analysis
,
9
,
61
77
.
Aoki
,
H.
(
2002
).
A complex-valued neuron to transform gray level images to phase information
. In
Proceedings of International Conference on Neural Information Processing
,
3
(pp.
1084
1088
).
Berlin
:
Springer
.
Aoki
,
H.
, &
Kosugi
,
Y.
(
2000
).
An image storage system using complex-valued associative memories
. In
Proceedings of International Conference on Pattern Recognition
,
2
(pp.
626
629
).
Piscataway, NJ
:
IEEE
.
Catoni
,
F.
,
Cannata
,
R.
, &
Zampetti
,
P.
(
2006
).
An introduction to commutative quaternions
.
Advances in Applied Clifford Algebras
,
16
,
1
28
.
de Castro
,
F. Z.
, &
Valle
,
M. E.
(
2017
).
Continuous-valued quaternionic Hopfield neural network for image retrieval: A color space study
. In
Proceedings of Brazilian Conference on Intelligent Systems
(pp.
186
191
).
de Castro
,
F. Z.
, &
Valle
,
M. E.
(
2020
).
A broad class of discrete-time hypercomplex-valued Hopfield neural networks
.
Neural Networks
,
122
,
54
67
.
Hitzer
,
E.
,
Nitta
,
T.
, &
Kuroe
,
Y.
(
2013
).
Applications of Clifford's geometric algebra
.
Advances in Applied Clifford Algebras
,
23
,
377
404
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2006
).
Fundamental properties of quaternionic Hopfield neural network
. In
Proceedings of International Joint Conference on Neural Networks
(pp.
1610
1615
).
Piscataway, NJ
:
IEEE
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2007
).
Dynamics of discrete-time quaternionic Hopfield neural networks
. In
Proceedings of International Conference on Artificial Neural Networks
(pp.
848
857
).
Berlin
:
Springer
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2008
).
Associative memory in quaternionic Hopfield neural networks
.
International Journal of Neural Systems
,
18
,
135
145
.
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2009
).
An iterative learning scheme for multistate complex-valued and quaternionic Hopfield neural networks
. In
Proceedings of International Joint Conference on Neural Networks
(pp.
1365
1371
).
Berlin
:
Springer
.
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2010
).
Commutative quaternion and multi-state Hopfield neural networks
. In
Proceedings of IEEE World Congress on Computational Intelligence
(pp.
1281
1286
).
Piscataway, NJ
:
IEEE
.
Isokawa
,
T.
,
Nishimura
,
H.
,
Saitoh
,
A.
,
Kamiura
,
N.
, &
Matsui
,
N.
(
2008
).
On the scheme of quaternionic multistate Hopfield neural network
. In
Proceedings of Joint 4th International Conference on Soft Computing and Intelligent Systems and 9th International Advanced Intelligent Systems
(pp.
809
813
).
Piscataway, NJ
:
IEEE
.
Jankowski
,
S.
,
Lozowski
,
A.
, &
Zurada
,
J. M.
(
1996
).
Complex-valued multistate neural associative memory
.
IEEE Transactions on Neural Networks
,
7
,
1491
1496
.
Kitahara
,
M.
, &
Kobayashi
,
M.
(
2014
).
Projection rule for rotor Hopfield neural networks
.
IEEE Transactions on Neural Networks and Learning Systems
,
25
,
1298
1307
.
Kobayashi
,
M.
(
2008
).
Pseudo-relaxation learning algorithm for complex-valued associative memory
.
International Journal of Neural Systems
,
18
,
147
156
.
Kobayashi
,
M.
(
2016
).
Gradient descent learning rule for complex-valued associative memories with large constant terms
.
IEEJ Transactions on Electrical and Electronic Engineering
,
11
,
357
363
.
Kobayashi
,
M.
(
2017
).
Quaternionic Hopfield neural networks with twin-multistate activation function
.
Neurocomputing
,
267
,
304
310
.
Kobayashi
,
M.
(
2018a
).
Twin-multistate commutative quaternion Hopfield neural networks
.
Neurocomputing
,
320
,
150
156
.
Kobayashi
,
M.
(
2018b
).
Hyperbolic Hopfield neural networks with directional multistate activation function
.
Neurocomputing
,
275
,
2217
2226
.
Kobayashi
,
M.
(
2020
).
Noise robust projection rule for hyperbolic Hopfield neural networks
.
IEEE Transactions on Neural Networks and Learning Systems
,
31
,
352
356
.
Kobayashi
,
M.
,
Yamada
,
H.
, &
Kitahara
,
M.
(
2011
).
Noise robust gradient descent learning for complex-valued associative memory
. In
Proceedings of the IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences
,
E94-A
(pp.
1756
1759
).
Tokyo
:
Maruzen
.
Krizhevsky
,
A.
(
2009
).
Learning multiple layers of features from tiny images
(Technical Report). University of Toronto
.
Lee
,
D. L.
(
2001
).
Improving the capacity of complex-valued neural networks with a modified gradient descent learning rule
.
IEEE Transactions on Neural Networks
,
12
,
439
443
.
Lee
,
D. L.
(
2006
).
Improvements of complex-valued Hopfield associative memory by using generalized projection rules
.
IEEE Transactions on Neural Networks
,
17
,
1341
1347
.
Minemoto
,
T.
,
Isokawa
,
T.
,
Nishimura
,
H.
, &
Matsui
,
N.
(
2016
).
Quaternionic multistate Hopfield neural network with extended projection rule
.
Artificial Life and Robotics
,
21
,
106
111
.
Muezzinoglu
,
M. K.
,
Guzelis
,
C.
, &
Zurada
,
J. M.
(
2003
).
A new design method for the complex-valued multistate Hopfield associative memory
.
IEEE Transactions on Neural Networks
,
14
,
891
899
.
Noest
,
A. J.
(
1988
).
Discrete-state phasor neural networks
.
Physical Review A
,
38
,
2196
2199
.
Tanaka
,
G.
, &
Aihara
,
K.
(
2009
).
Complex-valued multistate associative memory with nonlinear multilevel functions for gray-level image reconstruction
.
IEEE Transactions on Neural Networks
,
20
,
1463
1473
.
Valle
,
M. E.
(
2014
).
A novel continuous-valued quaternionic Hopfield neural network
. In
Proceedings of Brazilian Conference on Intelligent Systems
(pp.
97
102
).
Valle
,
M. E.
, &
de Castro
,
F. Z.
(
2018
).
On the dynamics of Hopfield neural networks on unit quaternions
.
IEEE Transactions on Neural Networks and Learning Systems
,
29
,
2464
2471
.
Zheng
,
P.
(
2014
).
Threshold complex-valued neural associative memory
.
IEEE Transactions on Neural Networks and Learning Systems
,
25
,
1714
1718
.