For a long time，the argument of “investigation of things”（格物）has been litigated. The fact that it has become the focus topic of scholars is inseparable from the upgrading of the “Four Books”（四书）led by the Great Learning（大学），therefore，scholars have been discussing it a lot. Since the 17th century，however，there has been a trend of “returning rites”（返礼学）thought in East Asia，which is often represented by opposition to Zhu Xi（朱熹）’s modification of Classics，and the ultimate destination of this trend points to the revival of the classical rites. But there is still a lack of in-depth research on this trend of thought，so this paper takes the theory of “extension of knowledge”（格物致知）as the entry point，through the analysis of Chen Qianchu（陈乾初），Yun Baekho（윤백호）and Ito Jinsai（伊藤仁斋）’s criticism of Zhu Xi’ annotations and even the Great Learning，so as to highlight the deconstruction process of the “Four Books” and also outline the general state of the trend of “returning rites” thought. An in-depth analysis of the trend of “returning rites” thought can not only inspire us to rethink the internal logic of the development form Neo-Confucianism to Evidential Scholarship（朴学），but also provides a new way to reflect on the “de-Neo-Confucianism” in the history of Confucianism in Joseon Dynasty and the “Ancient School” in the history of Confucianism in Edo period.