Abstract
The use of indefinite kernels has attracted many research interests in recent years due to their flexibility. They do not possess the usual restrictions of being positive definite as in the traditional study of kernel methods. This paper introduces the indefinite unsupervised and semi-supervised learning in the framework of least squares support vector machines (LS-SVM). The analysis is provided for both unsupervised and semi-supervised models, i.e., Kernel Spectral Clustering (KSC) and Multi-Class Semi-Supervised Kernel Spectral Clustering (MSS-KSC). In indefinite KSC models one solves an eigenvalue problem whereas indefinite MSS-KSC finds the solution by solving a linear system of equations. For the proposed indefinite models, we give the feature space interpretation, which is theoretically important, especially for the scalability using Nyström approximation. Experimental results on several real-life datasets are given to illustrate the efficiency of the proposed indefinite kernel spectral learning.
| Original language | English |
|---|---|
| Pages (from-to) | 144-153 |
| Number of pages | 10 |
| Journal | Pattern Recognition |
| Volume | 78 |
| DOIs | |
| Publication status | Published - Jun 2018 |
| Externally published | Yes |
Bibliographical note
Funding Information:The authors are grateful to the anonymous reviewer for insightful comments. The research leading to these results received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC AdG A-DATADRIVE-B (290923). This letter reflects only our views: The EU is not responsible for any use that may be made of the information in it. The research leading to these results received funds from the following sources: Research Council KUL: GOA/10/09 MaNet, CoE PFV/10/002 (OPTEC), BIL12/11T; PhD/Postdoc grants; Flemish Government: FWO: PhD /Postdoc grants, projects: G.0377.12 (Structured systems), G.088114N (Tensor based data similarity); IWT: PhD/Postdoc grants, projects: SBO POM (100031); iMinds Medical Information Technologies SBO 2014; Belgian Federal Science Policy Office: IUAP P7/19 (DYSCO, Dynamical systems, control and optimization, 2012–2017). Siamak Mehrkanoon was supported by a Postdoctoral Fellowship of the Research Foundation-Flanders (FWO). Xiaolin Huang is supported by National Natural Science Foundation of China (no. 61603248 ). Johan Suykens is a full professor at KU Leuven, Belgium. Siamak Mehrkanoon received the B.Sc. degree in pure mathematics and the M.Sc. degree in applied mathematics from the Iran University of Science and Technology, Tehran, Iran, in 2005 and 2007, respectively. He is holder of Ph.D. degrees in Numerical Analysis and Machine Learning from Universiti Putra Malaysia, Seri Kembangan, Malaysia, and KU Leuven, Belgium, in 2011 and 2015, respectively. He was a Visiting Researcher with the Department of Automation, Tsinghua University, Beijing, China, in 2014, a Postdoctoral Research Fellow with the University of Waterloo, Waterloo, ON, Canada, from 2015 to 2016, and a visiting postdoctoral researcher with the Cognitive Systems Laboratory, University of Tübingen, Tübingen, Germany, in 2016. He is currently an FWO Postdoctoral Research Fellow with the STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven. His current research interests include deep learning, neural networks, kernel-based models, unsupervised and semi-supervised learning, pattern recognition, numerical algorithms, and optimization. Dr. Mehrkanoon received several fellowships for supporting his scientific studies including Postdoctoral Mandate (PDM) Fellowship from KU Leuven and Postdoctoral Fellowship of the Research Foundation-Flanders (FWO). Xiaolin Huang received the B.S. degree in control science and engineering, and the B.S. degree in applied mathematics from Xi'an Jiaotong University, Xi’an, China in 2006. In 2012, he received the Ph.D. degree in control science and engineering from Tsinghua University, Beijing, China. From 2012 to 2015, he worked as a postdoctoral researcher in ESAT-STADIUS, KU Leuven, Leuven, Belgium. After that he was selected as an Alexander von Humboldt Fellow and working in Pattern Recognition Lab, the Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, where he was appointed as a group head. From 2016, he has been an Associate Professor at Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, China. In 2017, he has been awarded as “1000-Talent” (Young Program). His current research areas include machine learning, optimization, and their applications on medical image processing. Johan A.K. Suykens was born in Willebroek Belgium, on May 18, 1966. He received the M.S. degree in Electro-Mechanical Engineering and the Ph.D. Degree in Applied Sciences from the Katholieke Universiteit Leuven, in 1989 and 1995, respectively. In 1996 he has been a Visiting Postdoctoral Researcher at the University of California, Berkeley. He has been a Postdoctoral Researcher with the Fund for Scientific Research FWO Flanders and is currently a Professor (Hoogleraar) with KU Leuven. He is author of the books Artificial Neural Networks for Modelling and Control of Non-linear Systems (Kluwer Academic Publishers) and Least Squares Support Vector Machines (World Scientific), co-author of the book Cellular Neural Networks, Multi-Scroll Chaos and Synchronization (World Scientific) and editor of the books Nonlinear Modeling: Advanced Black-Box Techniques (Kluwer Academic Publishers) and Advances in Learning Theory: Methods, Models and Applications (IOS Press). Prof. Suykens received an IEEE Signal Processing Society 1999 Best Paper (Senior) Award and several best paper awards at international conferences. He was a recipient of the International Neural Networks Society 2000 Young Investigator Award for significant contributions in the field of neural networks. He has been awarded an ERC Advanced Grant 2011 and has been elevated IEEE Fellow 2015 for developing least squares support vector machine. In 1998, he organized an International Workshop on Nonlinear Modeling with Timeseries Prediction Competition. He served as an Associate Editor of the IEEE Transactions on Circuits and Systems from 1997 to 1999 and 2004 to 2007, and the IEEE Transactions on Neural Networks from 1998 to 2009. He served as a Director and an Organizer of the NATO Advanced Study Institute on Learning Theory and Practice, Leuven, in 2002, a Program Co-Chair of the International Joint Conference on Neural Networks in 2004 and the International Symposium on Nonlinear Theory and its Applications in 2005, an Organizer of the International Symposium on Synchronization in Complex Networks in 2007, a Co-Organizer of the Conference on Neural Information Processing Systems Workshop on Tensors, Kernels and Machine Learning in 2010, and the Chair of the International Workshop on Advances in Regularization, Optimization, Kernel methods and Support vector machines in 2013.
Publisher Copyright:
© 2018 Elsevier Ltd
Funding
The authors are grateful to the anonymous reviewer for insightful comments. The research leading to these results received funding from the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC AdG A-DATADRIVE-B (290923). This letter reflects only our views: The EU is not responsible for any use that may be made of the information in it. The research leading to these results received funds from the following sources: Research Council KUL: GOA/10/09 MaNet, CoE PFV/10/002 (OPTEC), BIL12/11T; PhD/Postdoc grants; Flemish Government: FWO: PhD /Postdoc grants, projects: G.0377.12 (Structured systems), G.088114N (Tensor based data similarity); IWT: PhD/Postdoc grants, projects: SBO POM (100031); iMinds Medical Information Technologies SBO 2014; Belgian Federal Science Policy Office: IUAP P7/19 (DYSCO, Dynamical systems, control and optimization, 2012–2017). Siamak Mehrkanoon was supported by a Postdoctoral Fellowship of the Research Foundation-Flanders (FWO). Xiaolin Huang is supported by National Natural Science Foundation of China (no. 61603248 ). Johan Suykens is a full professor at KU Leuven, Belgium. Siamak Mehrkanoon received the B.Sc. degree in pure mathematics and the M.Sc. degree in applied mathematics from the Iran University of Science and Technology, Tehran, Iran, in 2005 and 2007, respectively. He is holder of Ph.D. degrees in Numerical Analysis and Machine Learning from Universiti Putra Malaysia, Seri Kembangan, Malaysia, and KU Leuven, Belgium, in 2011 and 2015, respectively. He was a Visiting Researcher with the Department of Automation, Tsinghua University, Beijing, China, in 2014, a Postdoctoral Research Fellow with the University of Waterloo, Waterloo, ON, Canada, from 2015 to 2016, and a visiting postdoctoral researcher with the Cognitive Systems Laboratory, University of Tübingen, Tübingen, Germany, in 2016. He is currently an FWO Postdoctoral Research Fellow with the STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, KU Leuven. His current research interests include deep learning, neural networks, kernel-based models, unsupervised and semi-supervised learning, pattern recognition, numerical algorithms, and optimization. Dr. Mehrkanoon received several fellowships for supporting his scientific studies including Postdoctoral Mandate (PDM) Fellowship from KU Leuven and Postdoctoral Fellowship of the Research Foundation-Flanders (FWO). Xiaolin Huang received the B.S. degree in control science and engineering, and the B.S. degree in applied mathematics from Xi'an Jiaotong University, Xi’an, China in 2006. In 2012, he received the Ph.D. degree in control science and engineering from Tsinghua University, Beijing, China. From 2012 to 2015, he worked as a postdoctoral researcher in ESAT-STADIUS, KU Leuven, Leuven, Belgium. After that he was selected as an Alexander von Humboldt Fellow and working in Pattern Recognition Lab, the Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany, where he was appointed as a group head. From 2016, he has been an Associate Professor at Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, China. In 2017, he has been awarded as “1000-Talent” (Young Program). His current research areas include machine learning, optimization, and their applications on medical image processing. Johan A.K. Suykens was born in Willebroek Belgium, on May 18, 1966. He received the M.S. degree in Electro-Mechanical Engineering and the Ph.D. Degree in Applied Sciences from the Katholieke Universiteit Leuven, in 1989 and 1995, respectively. In 1996 he has been a Visiting Postdoctoral Researcher at the University of California, Berkeley. He has been a Postdoctoral Researcher with the Fund for Scientific Research FWO Flanders and is currently a Professor (Hoogleraar) with KU Leuven. He is author of the books Artificial Neural Networks for Modelling and Control of Non-linear Systems (Kluwer Academic Publishers) and Least Squares Support Vector Machines (World Scientific), co-author of the book Cellular Neural Networks, Multi-Scroll Chaos and Synchronization (World Scientific) and editor of the books Nonlinear Modeling: Advanced Black-Box Techniques (Kluwer Academic Publishers) and Advances in Learning Theory: Methods, Models and Applications (IOS Press). Prof. Suykens received an IEEE Signal Processing Society 1999 Best Paper (Senior) Award and several best paper awards at international conferences. He was a recipient of the International Neural Networks Society 2000 Young Investigator Award for significant contributions in the field of neural networks. He has been awarded an ERC Advanced Grant 2011 and has been elevated IEEE Fellow 2015 for developing least squares support vector machine. In 1998, he organized an International Workshop on Nonlinear Modeling with Timeseries Prediction Competition. He served as an Associate Editor of the IEEE Transactions on Circuits and Systems from 1997 to 1999 and 2004 to 2007, and the IEEE Transactions on Neural Networks from 1998 to 2009. He served as a Director and an Organizer of the NATO Advanced Study Institute on Learning Theory and Practice, Leuven, in 2002, a Program Co-Chair of the International Joint Conference on Neural Networks in 2004 and the International Symposium on Nonlinear Theory and its Applications in 2005, an Organizer of the International Symposium on Synchronization in Complex Networks in 2007, a Co-Organizer of the Conference on Neural Information Processing Systems Workshop on Tensors, Kernels and Machine Learning in 2010, and the Chair of the International Workshop on Advances in Regularization, Optimization, Kernel methods and Support vector machines in 2013.
Keywords
- Indefinite kernels
- Kernel spectral clustering
- Low embedding dimension
- Scalable models
- Semi-supervised learning