Before I lost my last hope I saw this post:
And I tried to take the suggestion and added the +tcp option into the dig command. Well, the server immediately delivers the correct answer this time. That implies the root cause is that I failed to open the UDP port 53! I checked the iptables and found that I only allowed the port 53 on TCP traffic.
So I added the following line to my iptables configuration file:
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
The world is in peace now. Forgive all my carelessness and ignorance please. However the real relief is that it again proves that every technical issue has a technical reason (and thus a technical solution).
Thank you all those who tried to help!