I do a lot of development in linux/unix. In fact, that’s where all my development is, but I spend a fair amount of time in the “CLI” or command line interface. I use putty and ssh in to edit files and look at logs and just generally do what I do.
Today I ran into a problem with a file and I couldn’t figure out what was wrong. I kept getting an error and the spacing was off so I started suspecting a hidden character. vi, the command line editor I use in linux can show you hidden characters. Just type:
And bingo, hidden characters are revealed.
Turning them back off is just as easy:
Write that down, it comes in handy.
Updated 2/25/10: Thanks to Stephen in the comments, I’ve updated the above commands. I had the : in the wrong place!
Greg Bergeson says
Thanks for this info on vi. I didn’t want to drag out the books and figure out which option it was again! I really needed this for file conversion between Mac OS and Windows OS (the infamous carriage return versus line feed issue). Thanks again!
You don’t need to post my response. I think you meant “:set list” and “:set nolist” instead of “set: list”. I got an error when I typed “set: list” and “:set: list”. Thought yuo might want to know so you can fix your post and save others the headache of figuring it out.
This post was very helpful never the less. Thanks!
Stephen, so right! Totally typo’d that. Updating it now.
Once you find the characters you want to remove you can use:
:%s///g to remove everything globally. The way to type those pesky special characters in vi (ie ^M) is to use ctrl-v then ctrl-m You then substitute the m for whatever letter you need.
Hope that helps!
Sorry the posting left out some of my characters. A sample of the above technique would be :%s/^M//g
Sorry for the mixup!
will take care of you dos file carry over ^M
Is there a way to show the hidden \t characters?
Also, if you like me would like a key-binding for this, you can use the
:set list!version. Notice the “!” at the end. This will toggle list on and off.
This is a very useful command. Thanks for sharing this.
We were having troubles running a service using a config file as the config file has hidden characters. I was able to find out the hidden characters in the file using this command and it resolved out issue.
Thanks very much!
Josh Lee says
I found out today that :set list in vim doesn’t actually always show dos characters, only sometimes. To make 100% sure you need to run “cat” on the file with the two flags “v” and “T”. So if your file is named foo.bar, you’d type in: “cat -vT foo.bar”
so much easier then getting up and going to my bookshelf to break out the o’reillys book. thanks
Thank you for the Vi tip. I have been pulling my hair out over some CSS that is not rendering in the browser. Here is the CSS:
Simple right?! The problem is the color is not registering, but the background is. But, when I delete the spaces between the newline and “color”. it works perfectly.
The CSS in question is CSS that I have cut and pasted from my browser into Vim.
When I use :set list, what I get is
I’m assuming ^I is tab and $ must be carriage return? Looks fine, right?
Do you know of any other way to view any special hidden chars?
I repeat, if I delete those few spaces, it works perfectly. So I know it’s not a CSS programming thing. This is totally bizarre.
I’ve run into this kind of problem maybe twice in the last 4 months. It’s a real PITA because the text is indeed correct and hunting down this kind of bug is tough…
Bradley Rentfrow says
This is what I found to removed the pesky ^M’s that mess up your script:
When you run cat -vT or run the script the ^M’s should be gone.
Basically it saves the file back into the UNIX format and scripts the DOS carriage returns from the file.
Thanks for the post. I now just have to decipher what I am seeing. Hmmm running :set list and then seeing some ^I or $ means what?
Deniz Gezmis says
If you happen to run Slackware Linux the following command should take care of those DOS related characters smoothly.
Michael Monteith says
On some systems the vi -b option works great. Here is a good note on the subject
Thanks for the post, it definitely help to save the reference, since I only use it like once in a while. :-)
Terry Mackie says
I was using diff and by eye balling the code, I couldn’t figure out why 2 lines were different. I used set list and saw the 2 lines each had a space at the end. I removed the spaces and diff was sucessful.
Hopping Bunny says
Helped me fix a strange problem that cropped up after I added some comments to a shell script !
Of course, if it is just the pesky ^M from a windows file that is the issue, there is always the good old “dos2unix” converston tool:
dos2unix -o a.xtx b.txt
That will automatically strip out the DOS line feed (carriage return code) stuff. HTH.
Brian S says
Another command to find odd and pesky unknown characters is to type “od -cb foobar” where foobar is your file name this will show the file with the character you see and the octal code for that character. The Octal Dump (od) program has many features and options, “man od” for more.
Brian, awesome tip! Thanks for sharing it. I need to compile some of the stuff people have added here to the post.
for a quick look at special chars, try this
the last character is an “ell” not a “one”
:1,$p prints the standard chars but not special
Mi Sa says
This is a great cheat sheet
Mi, that’s awesome! Thanks for sharing.
henry lai says
dos2unix -n windows.txt unix.txt
Sal Ferrarello says
I was just telling someone about this feature and I’m glad I found this post to share with them – thank you!
Additionally, you can define which hidden characters are displayed and how they appear.
I use Tim Pope’s Vim plugin vim-sensible, which sets the following.
set listchars=tab:>\ ,trail:-,extends:>,precedes:<,nbsp:+
Thanks again for this post!
Paola Simonian says
Hi! Thanks a lot for all the info!
I’m still trying to remove the $ that I’ve found… but I’ve tried :%s/$//g and didn’t work. What am I doing wrong?
Wow! 30 years of Unix/Linux and never knew that. My life just got slightly better.
Just what I needed a week ago before I rekeyed 4 script files. Now I KNOW I’m not crazy. No worries and and thanks to you that won’t happen again.