Almost every person you meet with aphasia will have lost most of their ability to name letters or numbers. There will be a disconnection between recognizing these numbers/letters ("Show me the 3") and naming them aloud ("What is this?"). Many people can accurately write numbers but cannot say them without writing them or without counting up from one ("one, two, three, FOUR") to answer "four".
Anomia, or problems with word-retrieval and naming upon demand, is one of the most persistent and universal characteristics of any type of aphasia (almost everyone has it and it takes the longest to get better). Numbers are part of language, so it's no surprise that aphasia extends to this concept. In terms of aphasia, numbers are no different from words.
Numbers can be more complex than words, however. For example, the number "3" is represented by an abstract shape and has a name "three". A number's shape has meaning only because we say it does. That's its identity—what it looks like in written form. We further say that a number represents a concept of quantity of something ('3 eggs"). A number "3" written by itself doesn't really mean anything.
We aren't born knowing these names and what they represent—we learn these as we go through school. Someone with aphasia can usually write the correct number but say something different. When you clarify "You said four (point to what they wrote), is that right?" If it is, everyone is good. If not, they can re-do it.
Some people with aphasia may get stuck in using finger gestures that are incorrect, which can be very confusing. If a client holds up two fingers when speaking, we can get off on the wrong track easily if I don't clarify that he means to say "2". Help them but showing them the correct number of fingers—some people may need you to physically put their fingers in that position at first.
Numbers tend to remain largely ignored in aphasia therapy because they are very "slippery", and because there are usually other skills that are more important at the time. Additionally, insurance isn't going to consider it an important aspect of language rehabilitation.
The good news is that numbers may be easier to re-learn than letters. Numbers make sense to retrain if it's something very important to you. If you can get by on writing or pointing to the numbers in the meantime, I'd use that strategy. To relearn numbers, you have to incorporate all modalities of speech and language—visual aspects, what it sounds like and writing it. It's going to take time and lots of practice, so get started!
