Multi-modality is a fundamental feature that characterizes biological systems and lets them achieve high robustness in understand- ing skills while coping with uncertainty. Relatively recent studies showed that multi-modal learning is a potentially eective add-on to articial systems, allowing the transfer of information from one modality to an- other. In this paper we propose a general architecture for jointly learn- ing visual and motion patterns: by means of regression theory we model a mapping between the two sensorial modalities improving the perfor- mance of articial perceptive systems. We present promising results on a case study of grasp classication in a controlled setting and discuss future developments.