Dear Chenxiao,

Thank you very much for your question and your interest in my post. For your question, yes, both maximum likelihood and mean square error can be used to estimate the GP parameters. As for which method is better, I would like to refer you to this paper:

F. Bachoc, Cross Validation and Maximum Likelihood estimations of hyper-parameters of Gaussian processes with model misspecification, 2013.

[https://arxiv.org/abs/1301.4320]

In the paper, the author discussed the pros and cons of using different parameter estimation methods in different situations.

Hope that paper could be helpful for your work. Good luck!

Best,

Shuai