Inelastic scattering of electrons in a crystalline environment may be represented by a complex non-hermitian potential. Completed generalised expressions for this inelastic electron scattering potential matrix, including virtual inelastic scattering, are derived for outer-shell electron and plasmon excitations. The relationship between these expressions and the general anisotropic dielectric response matrix of the solid is discussed. These generalised expressions necessarily include the off-diagonal terms representing effects due to departure from translational invariance in the interaction. Results are presented for the diagonal back structure dependent inelastic and virtual inelastic scattering potentials for Si, from a calculation of the inverse dielectric matrix in the random phase approximation. Good agreement is found with experiment as a function of incident energies from 10 eV to 100 keV. Anisotropy effects and hence the interaction de localisation represented by the off-diagonal scattering potential terms, are found to be significant below 1 keV. 38 refs., 2 figs.