We prove recursive formulas for the Taylor coefficients of cusp forms, such as Ramanujan's Delta function, at points in the upper half-plane. This allows us to show the non-vanishing of all Taylor coefficients of Delta at CM points of small discriminant as well as the non-vanishing of certain Poincaré series. At a "generic" point, all Taylor coefficients are shown to be non-zero. Some conjectures on the Taylor coefficients of Delta at CM points are stated.