Defect levels in Mg-doped GaN epitaxial layers, grown on sapphire substrates by metal organic chemical vapor deposition (MOCVD), were investigated using high-resolution photoinduced transient spectroscopy (HRPITS). A novel approach to extraction of trap parameters from the photocurrent relaxation waveforms recorded in a wide temperature range has been applied. It is based on the two-dimensional analysis of the waveforms as a function of time and temperature using the correlation procedure or inverse Laplace algorithm. The effect of annealing on the material electrical properties and defect structure has been studied. Before annealing, the layer resistivity at 300 K was ~5x105 Ωcm and after a heat treatment at 780 oC it dramatically dropped to ~2.5x102 Ωcm. In the as-grown layer with the high-resistivity, six traps with activation energies of 0.12, 0.13, 0.66, 0.95, 1.32 and 1.52 eV were revealed. The 0.12-eV and 0.13-eV traps are tentatively attributed to the carbon atoms occupying Ga sites in the vicinity of dislocations and in the dislocation-free regions, respectively. The 0.66-eV trap can be assigned to the complex composed of a nitrogen vacancy and hydrogen atom (VN -H). The 0.95-eV trap is likely to be attributed to the complex involving gallium vacancy and two hydrogen atoms (VGa -2H) and the 1.32-eV trap seems to be related to the carbon interstitial Ci. In the sample subjected to the heat treatment, twelve traps with activation energies ranging from 0.12 to 1.38 eV were detected. In particular, the 0.17-eV trap related to the MgGa acceptor, as well as the traps with activation energies of 0.59 eV, 1.06 eV and 1.22 eV assigned to MgGa-VN , VGa-ON and Ci-Cs, complexes, respectively, were found. The results indicate that the main mechanism leading to electrical activation of Mg atoms is the decomposition of neutral MgGa-H complexes. On the other hand, the annealing results in a self-compensation through the formation of deep-level complex defects.