Recent breakthroughs in generative models have sparked unprecedented interest in a wide range of downstream applications, particularly in the realm of augmented and virtual reality, where immersive content creation is paramount. While significant progress has been made in generating high quality images and videos, creating coherent panoramic videos still remains an open challenge. In this work, we introduce VideoPanda, a novel approach for synthesizing 360-degree videos conditioned on text or video data. VideoPanda leverages multi-view attention layers to augment a video diffusion model, enabling it to be multi-view aware and capable of generating immersive panoramic content. Our model is jointly trained using three conditions: text-only, single-view video and single-view video with multi-view images. This unified training allows for video synthesis from text and video as well as long video generation using the same model trained once. Extensive evaluations on the WEB360 dataset, demonstrate that VideoPanda generates more realistic and coherent 360-degree panoramas across all input conditions compared to existing methods.