CDA Computational Design and Adaptation

Liquid State Machine Papers

I've done a bit with echo state networks (ESNs) and mostly neglected the more neuroscience motivated liquid state machine (LSM) research. However, there is some great work going on led by W. Maass. Here are two stand-out publications:

Distributed Fading Memory for Stimulus Properties in the Primary Visual Cortex - actual data from cat neocortex showing reservoir computing sort of responses.

State-dependent computations: spatiotemporal processing in cortical networks - fascinating review of a large body of research including nice discussions of high-dimensional trajectories.

Paper: Recent Advances in Hierarchical Reinforcement Learning (2003)

I came across this article today in my search for work on hierarchical reinforcement learning:
Recent Advances in Hierarchical Reinforcement Learning by Barto and Mahadevan, 2003. While much of the paper is on Dietterich's MAXQ and Parr and Russell's HAM algorithms, there is a section on hierarchical memory that looks interesting and seems closer to my area of interest than the MAXQ algorithm.

I'm just beginning my exploration of the hierarchical memory concept, thanks to Barto and Mahadevan's paper. Please post if you have good references that are even more recent than 2003.

Peter

Book: “Reinforcement Learning: An Introduction” by Sutton and Barto

Sutton and Barto introduce reinforcement learning in their clear, comprehensible book. Like many textbooks that are meant to be able to be used in pieces, it suffers from a bit of redundancy. However, it does well on a number of fronts.

Comparing Echo State Networks with Multiple Context

Echo State Networks (ESN) take advantage of the computing power now available, while Multiple Context (MC), the basic learning system for the PURR-PUSS robot brain, was developed when computing power was minimal. This suggests to me that the design of MC should be able to benefit from that of ESN. I will be talking only about ESN used for prediction

Both MC and ESN are driven by a complex short-term memory. In each case, there is a strong in-built structure. In an ESN it is a recurrent neural network with pre-set connection strengths. In MC it is a collection of contexts of events, where the event types of the events in a context are prescribed by fixed templates.

In the ESN, prediction is improved by a set of variable connections that pick out (in a mysterious way!) the appropriate parts of the recurrent network to make the prediction.    

 In the MC, prediction is improved by accumulating experienced contexts that have predicted successfully in the past. It allows one-off learning as well as gradual learning. After a time, some of these are discarded because of  poor performance and shortage of space.

The MC could be made more like the ESN if it began with a very large number of templates and had a variable weight attached to each template that measured the usefulness of  predictions coming from the contexts of that template. The weights would be taken into account when contexts from different templates were predicting different events. They could lead to the discarding of templates and even the introduction of new ones.

 The ramifications of this idea are considerable!

John H Andreae

Langford on Research

Here's a nice posting from John Lagford on pursuing grand ML challenges.

Necessary and Sufficient Research

Researchers are typically confronted with big problems that they have no idea how to solve. In trying to come up with a solution, a natural approach is to decompose the big problem into a set of subproblems whose solution yields a solution to the larger problem. This approach can go wrong in several ways.

1. Decomposition failure. The solution to the decomposition does not in fact yield a solution to the overall problem.
2. Artificial hardness. The subproblems created are sufficient if solved to solve the overall problem, but they are harder than necessary.

Then later he lists some things he thinks are necessary to achieve, "a master machine learning algorithm that can solve any reasonable learning problem where “reasonable” includes at least the set that humans can solve."

1. Large data
2. Online learning
3. Interactive learning
4. Compositional design
5. Large contexts
6. Nonlinearity

Read the post to see what these qualities mean in detail.  I agree with all of them, but I haven't done a lot to pursue #1 yet.