Mark B. Ring, Ph.D.

I am an external advisor at IDSIA in Lugano, Switzerland, working with Juergen Schmidhuber, particularly on the IM-Clever project on topics related to hierarchical reinforcement learning.

Last update: February 27, 2012

Research Interests:

My research tends to revolve around a single focus: Continual Learning—the unending quest to learn and develop, to comprehend and interract with the world increasingly well, and thus to achieve reward increasingly well. My dissertation, Continual Learning in Reinforcement Environments, explored this and related issues in depth. Although many ideas discussed in the dissertation have recently become increasingly fashionable, at the time of their publication, much of the work was breaking new ice in uncharted waters.

The Temporal Transition Hierarchies (TTH) algorithm, when seen from the current moment looking backwards, was the first temporal function approximator that intelligently and incrementally increased history length to resolve contradictions. When seen from the current moment looking forward, TTH's form expectation hierarchies, and as such represent a (perhaps unwitting) predecessor to predictive state representations (PSRs) in that they explicitly encode all future action and observation trajectories as contingencies over intervening actions and observations. The algorithm, which was also presented as an incremental method for learning FSAs (NIPS 5, 1992), was first and foremost intended for use in reinforcement environments (SAB 2, 1992).

Papers:

2011
: [pdf] Mark Ring, Tom Schaul, Jürgen Schmidhuber. The Two-Dimensional Organization of Behavior. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. Abstract | Bibtex | pdf; Hung Ngo, Mark Ring, Jürgen Schmidhuber. Curiosity Drive based on Compression Progress for Learning Environment Regularities. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epignetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. online; [pdf] Matt Luciw, Vincent Graziano, Mark Ring, Jürgen Schmidhuber. Artificial Curiosity with Planning for Autonomous Visual and Perceptual Development. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring. Recurrent Transition Hierarchies for Continual Learning: A general overview. In Proc. Joint IEEE International Conference on Development and Learning (ICDL) and on Epigenetic Robotics (ICDL-EpiRob 2011), Frankfurt, 2011. Abstract | Bibtex | pdf; [pdf] Laurent Orseau, Mark Ring. Self-Modification and Mortality in Artificial Agents. AGI, 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring, Laurent Orseau, Delusion, Survival, and Intelligent Agents. AGI, 2011. Abstract | Bibtex | pdf; [pdf] Leo Pape, Faustino Gomez, Mark Ring, Jürgen Schmidhuber. Modular deep belief networks that do not forget. International Joint Conference on Neural Networks (IJCNN-2011, San Francisco), 2011. Abstract | Bibtex | pdf; [pdf] Mark Ring, Tom Schaul. Q-error as a Selection Mechanism in Modular Reinforcement-Learning Systems. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-2011, Barcelona), 2011. Abstract | Bibtex | pdf; [pdf] Sun Yi, Faustino Gomez, Mark Ring, Jürgen Schmidhuber. Incremental Basis Construction from Temporal Difference Error. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011. Abstract | Bibtex | pdf

2005

[pdf] Eddie J. Rafols, Mark B. Ring, Richard S. Sutton, Brian Tanner, Using Predictive Representations to Improve Generalization in Reinforcement Learning, Proceedings of the 19^th International Joint Conference on Artificial Intelligence, 2005. Abstract | Bibtex

[pdf] Draft: Toward a formal framework for Continual Learning, post-NIPS workshop on Inductive Transfer, 2005.

1997; [pdf] Mark Ring. RCC Cannot Compute Certain FSA, Even with Arbitrary Transfer Functions, from Advances in Neural Information Processing Systems 10 (NIPS 10), 1997. Abstract | Bibtex
[pdf] [ps] [ps.Z]
: [pdf] Mark Ring. CHILD: A First Step Towards Continual Learning, Machine Learning Journal, vol. 28, 1997. Also appears as Chapter 11 in Learning to Learn, S. Thrun and L. Pratt, editors. Abstract | Bibtex
[pdf] [ps] [ps.gz]; 1996; [pdf] Mark Ring. Finding Promising Exploration Regions by Weighting Expected Navigation Costs, GMD Technical Report, Arbeitspapiere der GMD 987, April, 1996. Abstract | Bibtex
[pdf] [ps] [ps.Z]; 1995; [pdf] Mark Ring. Finding promising exploration regions by weighting expected navigation costs, working notes for talk given at the AAAI symposium on Active Learning, 1995. Abstract
[pdf] [ps] [ps.Z]; 1994; [pdf] Mark Ring. Continual Learning in Reinforcement Environments, University of Texas at Austin dissertation, 1994. See my dissertation page for more information. Abstract | Bibtex
[pdf]; 1993; [pdf] Mark Ring. Sequence Learning with Incremental Higher-Order Neural Networks, University of Texas at Austin AI lab technical report, 1993. Abstract | Bibtex
[pdf] [ps] [ps.Z]; 1992; [pdf] Mark Ring. Learning Sequential Tasks by Incrementally Adding Higher Orders, from Advances in Neural Information Processing Systems 5 (NIPS5), 1993. Abstract | Bibtex
[pdf] [ps] [ps.Z]

[pdf] Mark Ring. Two Methods for Hierarchy Learning in Reinforcement Environments, in From Animals to Animats 2: Proceedings of the Second International Conference on Simulation of Adaptive Behavior (SAB '92), proceedings dated 1993. Abstract | Bibtex

[pdf] [ps] [ps.Z]

1991

Mark Ring. Incremental Development of Complex Behaviors through Automatic Construction of Sensory-motor Hierarchies, from the proceedings of the Eighth International Workshop (ML91), 1991. Abstract | Bibtex

[ps] [ps.Z]

Contact Information

Mark B. Ring, Ph.D.

683 S. Glenhurst Dr.

Anaheim Hills, CA 92808

Phone: (714) 974-0123

Email: "ring" (use host: cs.utexas.edu)

Mark B. Ring