Let me preface this by saying, I am extremely fortunate to be working a fully remote job where I am able to manage my environment and generally be as I am without too much trouble from that.
It’s, of course subject to changes and the whims of my ‘leaders’ but its pretty great for now, and has been for five years.
I have been to the office exactly twice. Once to pick up my equipment, and once for a two-day all staff social that took me a week to recover from.
My new boss wants to have quarterly in-office strategic cheerleading sessions. I cannot imagine many scenarios where I will feel my disability more, and do not want to go. Not just because of the discomfort, but because I understand the setback that will come from my team seeing me spaced out and weird. Because I know I will not be able to participate the way my new boss desires, and I need to figure out how to regain the standing that I am sure to lose from this ‘day of connecting with others’
I would check to see what the process is for requesting workplace accommodations at your company and then see if you can get your doctor to provide the appropriate paperwork saying that you cannot participate in large group activities or that you need to be 100% remote.