Security is becoming an essential problem for integrated circuits (ICs). Various attacks, such as reverse engineering and dumping on-chip data, have been reported to undermine IC security. IEEE 1149.1, also known as JTAG, is primarily used for IC manufacturing test but inevitably provides a “backdoor” that can be exploited to attack ICs. Encryption has been used extensively as an effective mean to protect ICs through authentication, but a few weaknesses subsist, such as key leakage. Signature-based techniques ensure security using a database that includes known attacks, but fail to detect attacks that are not contained by the database. To overcome these drawbacks, a two-layer learning-based protection scheme is proposed. Specifically, the scheme monitors the execution of JTAG instructions and uses support vector machines (SVM) to identify abnormal instruction sequences. The use of machine learning enables the detection of unseen attacks without the need for key-based authentication. The experiments based on the OpenSPARC T2 platform demonstrate that the proposed scheme improves the accuracy of detecting unseen attacks by 50% on average when compared to previous work.