Using information entropy bounds to design VLSI friendly neural networks

In recent years, neural networks have consolidated as a valid paradigm able to compete successfully with classical approaches. VLSI implementations of such networks would offer a higher reliability, a better speed and a smaller size. However, the cost of such implementations is very sensitive to factors like the precision used for the weights and the complexity of the network Although it has been shown very recently that VLSI friendly neural networks (i.e. networks using integer weights in a very, restricted range) can solve an), problem if the weight range is chosen appropriately, there are no known algorithms or heuristics for designing such networks. This paper presents a method for calculating the minimal size of a VLSI optimal network for a given problem. A VLSI optimal network is a network using integer weights in a given range [-p,p] and units with a small constant fan-in. The value p is a small integer which is calculated from the problem parameters such that the problem is guaranteed to have a solution. It is shown that the number of weights can be lower bounded in the worst case by the expression: m(n+1[nlog(d(max)/d(min)) + (n-1)log (1/d(min)+1) + 1/2log(n-1) - loge/12 . (n-1) + c] where d(min) is the minimum distance between patterns of opposite classes, d(max) is the maximum distance between any patterns, m is the number of patterns in the largest class, n is the number of dimensions and c is a constant. The methodology is rested on various problems using a limited precision integer weights constructive algorithm.