Young radio AGN are pivotal for our understanding of many of the still-debated aspects of AGN feedback. In this paper we present a study of the interstellar medium (ISM) in the compact, peaked-spectrum radio galaxy PKS B1934-63 using X-shooter observations. Most of the warm ionized gas resides within a circum-nuclear disk with a radius of about 200 pc that is likely to constitute the gas reservoir from which the central black hole feeds. On the other hand, we find a biconical outflow of warm ionized gas with an estimated radius of 59 +/- 12 pc. This matches the radial extent of the radio source and suggests that the outflow is jet driven. Thanks to the superior wavelength coverage of the data, we can estimate the density of the warm ionized gas using the trans-auroral line technique, and we find that the outflowing gas has remarkably high density, up to log n(e) (cm(-3)) similar or equal to 5.5. The estimated mass outflow rate is low (M = 10(-3) 10(-1) M Theta yr(-1)), and the AGN feedback operates at relatively low efficiency (/L-bol similar to 10(-4) - 10(-3)%). In addition, optical and near-IR line ratios show that the expansion of the radio source drives fast shocks (with velocities v(s) >= 500 km s(-1)) that ionize and accelerate the outflowing gas. At odds with the properties of other compact, peaked-spectrum radio sources hosting warm ionized gas outflows, we do not find signs of kinematically disturbed or outflowing gas in phases colder than the warm ionized gas. We argue that this is due to the young age of our source and thus to the recent nature of the AGN-ISM interaction, and suggest that cold gas forms within the outflowing material and the shock-ionized outflowing gas of PKS B1934-63 did not have enough time to cool down and accumulate in a colder phase. This scenario is also supported by the multi-phase outflows of other compact and young radio sources in the literature.