1.
When the internet was first created, the people who made it may not have been thinking of everyone when they were coming up with it. It was for academics first and foremost - a class of people who were required to have a computer to access the network in the first place.
But times change. The world changes. We don't live in an age of digital elitism, any more - everyone has a computer in their pocket, even if people want to turn up their noses and act like phones are inferior. Yes, smartphones suck, for any variety of reasons. They're inconvenient to have to write extra code for. Many people find themselves having issue with the complete and utter consolidation of everything onto one device.
2.
But just because you hate something doesn't mean it's a reason to exclude people on basis. Adding a snippet of @media max-width(xxx) and modifying things that break on smaller screen sizes isn't difficult - it's just a bit time consuming. Besides - wouldn't it feel rewarding to make your site smooth and comprehensible on as many platforms and sizes as possible? Fulfilling to expand the radius of your knowledge?
We live in a time where more people use phones than computers - for a dozen reasons, it's not right nor fair to exclude them because you don't feel like it or you don't know how. Just as I said before - to learn is to love. That includes learning how to expand your code to make your website more accessible to more people. It does matter - people might be on a phone for any dozens of reasons.
3.
Personally? I have to spend a lot of time at work, in a kitchen. I can't use a computer there, but I have a surprising amount of downtime. I don't like nor want to use social media - I don't like getting irritated and pissed off, especially at work.
I would much rather surf the web and explore more websites and spend my time reading peoples' thoughts, and it's frankly a bummer when peoples' sites are completely broken and unnavigable on mobile.
4.
There are people in worse situations than I am, and I don't think they deserve to be left behind or left out in this era of modern technology. Now more than ever should be an age of accessibility - even if it's not pretty, it should still be functional. Someone might be stuck on the street and not have a computer, or at least nowhere to keep one; they could be bedbound, and unable to sit at a desk.
All of these people still deserve to be included in the indie web, and they deserve to see what we have to share. I may not know about any ways to create a website from your phone other than something like Carrd, but that doesn't mean they can't look around and be inspired and dream of something better.
5.
We all want to escape the modern web. Or, at least, most of us do. People have gotten sick of being exploited and enraged just to pad the pockets of the richest people on the planet, and everyone deserves a space on the indie web. Do your part. Take a look at your site on mobile, or shrink your browser. Test, and make it better. Make bad code good, and make good code better. Follow the footsteps of the founders of the internet.
After all, how can we communicate when half the people on Earth can't even read what you're trying to say?