Afaik no, I think currently in literature bounding boxes are "by definition" rectangles, more complex shapes start approaching segmentation, which detectron does support however. From the segmetations you could then derive shapes like quadrilateral bounding boxes, not sure if e.g. OpenCV would support that.