In this work we studied several families of learning algorithms, including Support Vector Machines, Multiple Kernel Learning and metric learning. We focused on two fundamental tasks in Machine Learning which are classification and feature selection. We derived novel learning algorithms for classification and for feature selection, which outperformed the state-of-the-art in several experiments. Moreover, we discovered the relationship among algorithms and built a unified framework of various learning algorithms, more precisely, a unified framework of Support Vector Machines, Multiple Kernel Learning and metric learning.