Bohebbian Rhapsody

Douglas Eck (University of Montreal)
Bram Bakker (IDSIA, Lugano Switzerland)
Freddie Mercury (Queen's University)


Is this the real life
Is this just fantasy
Caught in a state space
No escape from dimensionality
Open your gates
Look up to the weights and see
I'm just a poor boy, i need no sympathy
Because I'm temporal error flow
My system is dynamical
Anyway the error flows doesn't really matter to me to me.

Yoshua just killed a net    [Also Sepp]
He showed decaying gradient  [Also Sepp]
Over time the error's spent  
Yoshua, life had just begun   
And now I'm in a local minimum

Juergen Schmidh--ooooo--ber,
Didn't mean to make you cry
If I don't converge again this time tomorrow
Try to guess all my weights, as if nothing really matters
Too late, my time has come,
Divergence won't abate
Activations saturate

Goodbye everybody, I've got to go
Gonna change my name to something with ``kernel'' in it
Mama ooo I don't want to die,
I sometimes wish I'd had the name ``manifold''
(Loopy nets?  Recurrent Control? . . . Oh it doesn't really matter)

I see a little silhouetto of a man,
Paul Werbos Paul Werbos will you do a tutorial?
Thunderbolt and lightning very very frightening me
Zimmermann, Pearlmutter,
Hochreiter, John Moody,
P. Tino Figaro Magnifico

But I'm just a poor boy and nobody loves me
He's just a network very untrendy
We do not care about sequentiality!

Easy come easy go, will you let me flow?
BAYESmillah! no, we will not let you flow (let him flow)
BAYESmillah! no, we will not let you flow (let him flow)
Will not let you flow (let me flow)
Will not let you flow  (let me flow)
No,no,no,no,no,no,no-

Mama mia, mama mia, mama mia let me flow
Marvin Minsky has a devil put aside for me,for me,for me-

So you think you'll learn sequences without memory?
So its only reduction of dimensionality?
Well maybe. But don't forget speech, robots and finance
And then theres control, and cars made by Ford, and there's music

It's memory that matters
The world is POMDP
Gradients may vanish
But sequence learning is the place to be

Anyway,  the error flows....