From neural networks to the Category of composable supervised learning algorithms in Scala with compile-time matrix checking based on singleton-types
Neurocat is an experimental toy library studying 2 things: - The link between category theory and supervised learning algorithm and neural networks through the concepts described in this amazing paper Backprop as Functor: A compositional perspective on supervised learning which tries to unify (partly at least) category theory & supervised learning concepts & simple neural networks/
3as the value
3but also the type
3. My matrix experimentations use the little & really cool library singleton-ops by Frank S. Thomas the author of the great
refinedlibrary too. Shapeless
Natis a nice idea but not good for big naturals because this is a recursive structure (down to 0) checked at compile-time so
100000as
Very superficially, the idea of the paper is quite simple (minus a few details): - A supervised learning algorithm can be seen as a structure able to approximate a function
A -> Brelying on parameters
Pwhich are updated through an optimization/training process using a set of training samples. - This paper shows that the set of supervised learning algorithms equipped with 3 functions (implement, update-params, request-input) forms a symmetric monoidal category
Learnand then demonstrates that supervised learning algorithms can be composed - It also shows that there exists a Functor from the category
ParaFnof parametrised functions
P -> A -> Bto
Learncategory.
ParaFn -> Learn
InputLayer -> OutputLayerparametrised by the weights
W.
I: NNet -> ParaFn
NNet -> Learn : (ParaFn -> Learn) ∘ (NNet -> ParaFn)
I'll stop there for now but my work has just started and there are more concepts about the bimonoidal aspects of neural networks under euclidean space constraints and pending studies about recurrent networks and more.
Discovering that formulation, I just said: "Whoaaa that's cool, exactly what I had in mind without being able to put words on it".
Why? Because everything I've seen about neural networks looks like programming from the 70s, not like I program nowadays with Functional Programming, types & categories.
This starts unifying concepts and is exactly the reason of being of category theory in maths. I think programming learning algorithms will change a lot in the future exactly as programming backends changed a lot those last 10 years.
I'm just scratching the surface of all of those concepts. I'm not a NeuralNetwork expert at all neither a good mathematician so I just want to open this field of study in a language which now has singleton-types allowing really cool new ways of manipulating data structures
So first, have a look at this sample:
For info, to manipulate matrices, I used ND4J to have an array abstraction to test both in CPU or GPU mode but any library doing this could be used naturally.