As someone who uses liquid nitrogen as part of their daily routine (in a lab) and who has made liquid nitrogen ice cream on several occasions, my advice would be to not use a plastic container to hold liquid nitrogen unless it's designed to.
Yes, the plastic vessel may become brittle and may fracture. I have seen some materials shatter with extreme violence when frozen with liquid nitrogen; do not attempt this. Vessels for handling cryogenic liquids are designed especially for the job, and made from materials such as stainless steel, styrofoam, teflon, HDPE or silvered borosilicate glass. Unless you know with certainty that your container is designed to withstand temperatures of -196 °C (-321 °F), do not attempt this procedure.