A M4A file is a file containing audio in compressed with MPEG 4 technology, which is a lossy compression algorithm, similar to MP3 one, but improved to get better quality in lesser space.

If you are wandering about why they didn't simply used .MP4 extension, the answer is easy: .MP4 extension is used for MPEG 4 Audio files as well, but it is also used for videos, since MPEG 4 standard also includes video compression. Thus M4A basically means MPEG 4 Audio and make easier to distinguish audio files from video files.