In this paper, a brain-inspired method is proposed for real-time facial expression recognition. The brain emotional learning(BEL) model mimics the high speed of the emotional learning mechanism in brain, it has the superior feature of fast learning and it can avoid slow convergence in traditional networks. To improve the performance of BEL model, particle swarm optimization(PSO) is applied to optimize the weights of BEL model. The integrated classifier named as PSO-BEL is tested on the well-known Cohn-Kanade database, JAFFE database, the average recognition rates reach 97.71% and 96.77%, respectively. Moreover, a real-world interactive interface is established. The comparisons of experiments with the traditional methods indicate the superiority of the proposed PSO-BEL in terms of recognition accuracy and computing time. It demonstrates that the proposed PSO-BEL can successfully meet the requirement of real-time facial expressions recognition.