Sending HTML to a shortcode, issues with quotes and safe filter

I have a code shortcode, which basically wraps my code in a custom <code>. Now, when supplying HTML, the double quotes mess with Tera templates, so they don’t work. Using the escape sequence &quot; did solve that issue, now the escape sequences don’t get resolved unless I use the safe filter when displaying my code inside the custom <code>. I thought using safe would do the opposite, like not escape & de-escape my text. Am I misunderstanding?