Rapid and accurate prediction of molecular properties is a fundamental task in drug discovery. In recent years, deep learning-based molecular property prediction methods have received much attention and recent successes have shown that learning the representations of molecular structures by applying graph neural networks (GNNs) can achieve better prediction results. However, most previous approaches typically focus on learning atomic embedding, while in this paper, we propose a novel attention method based on atom pair embedding, and it was applied to two types of prediction task. Firstly, learning of atom pair embedding was done on 2D molecular graphs for predicting a series of ligand properties and secondly, the atom pair embedding was learned on ligand/protein 3D complex structures together with axial attention network to predict protein-ligand interaction. In MolecularNet benchmark datasets, our method achieved better performance than previous state-of-the-art models in ten property prediction tasks and in the task for protein-ligand interaction prediction, our method also obtained superior results on the PDB2016 dataset than a collection of reference models. Our source code will be publicly available upon the acceptance of the manuscript.