I do a lot of development in linux/unix. In fact, that’s where all my development is, but I spend a fair amount of time in the “CLI” or command line interface. I use putty and ssh in to edit files and look at logs and just generally do what I do.
Today I ran into a problem with a file and I couldn’t figure out what was wrong. I kept getting an error and the spacing was off so I started suspecting a hidden character. vi, the command line editor I use in linux can show you hidden characters. Just type:
:set list
And bingo, hidden characters are revealed.
Turning them back off is just as easy:
:set nolist
Write that down, it comes in handy.
Updated 2/25/10: Thanks to Stephen in the comments, I’ve updated the above commands. I had the : in the wrong place!
Thanks for this info on vi. I didn’t want to drag out the books and figure out which option it was again! I really needed this for file conversion between Mac OS and Windows OS (the infamous carriage return versus line feed issue). Thanks again!
You don’t need to post my response. I think you meant “:set list” and “:set nolist” instead of “set: list”. I got an error when I typed “set: list” and “:set: list”. Thought yuo might want to know so you can fix your post and save others the headache of figuring it out.
This post was very helpful never the less. Thanks!
Stephen, so right! Totally typo’d that. Updating it now.
Once you find the characters you want to remove you can use:
:%s///g to remove everything globally. The way to type those pesky special characters in vi (ie ^M) is to use ctrl-v then ctrl-m You then substitute the m for whatever letter you need.
Hope that helps!
Sorry the posting left out some of my characters. A sample of the above technique would be :%s/^M//g
Sorry for the mixup!
:%s/.$//g
will take care of you dos file carry over ^M
cheers\r
Is there a way to show the hidden \t characters?
Thanks,
Also, if you like me would like a key-binding for this, you can use the
:set list!version. Notice the “!” at the end. This will toggle list on and off.This is a very useful command. Thanks for sharing this.
We were having troubles running a service using a config file as the config file has hidden characters. I was able to find out the hidden characters in the file using this command and it resolved out issue.
Thanks very much!
I found out today that :set list in vim doesn’t actually always show dos characters, only sometimes. To make 100% sure you need to run “cat” on the file with the two flags “v” and “T”. So if your file is named foo.bar, you’d type in: “cat -vT foo.bar”
so much easier then getting up and going to my bookshelf to break out the o’reillys book. thanks
Thank you for the Vi tip. I have been pulling my hair out over some CSS that is not rendering in the browser. Here is the CSS:
.quiz-button-go:hover{
color:#ddffff;
background:#28e07e;
}
Simple right?! The problem is the color is not registering, but the background is. But, when I delete the spaces between the newline and “color”. it works perfectly.
The CSS in question is CSS that I have cut and pasted from my browser into Vim.
When I use :set list, what I get is
44.quiz-button-go:hover{$
45 $
46 ^Icolor:#ddffff;$
47 ^Ibackground:#28e07e;$
48 }$
I’m assuming ^I is tab and $ must be carriage return? Looks fine, right?
Do you know of any other way to view any special hidden chars?
I repeat, if I delete those few spaces, it works perfectly. So I know it’s not a CSS programming thing. This is totally bizarre.
I’ve run into this kind of problem maybe twice in the last 4 months. It’s a real PITA because the text is indeed correct and hunting down this kind of bug is tough…
This is what I found to removed the pesky ^M’s that mess up your script:
:e ++ff=dos
:setlocal ff=unix
:w
When you run cat -vT or run the script the ^M’s should be gone.
Basically it saves the file back into the UNIX format and scripts the DOS carriage returns from the file.
Thanks for the post. I now just have to decipher what I am seeing. Hmmm running :set list and then seeing some ^I or $ means what?
If you happen to run Slackware Linux the following command should take care of those DOS related characters smoothly.
fromdos output_file.txt
On some systems the vi -b option works great. Here is a good note on the subject
http://www.devdaily.com/blog/post/linux-unix/vi-editor-binary-mode-remove-control-m-characters
Thanks for the post, it definitely help to save the reference, since I only use it like once in a while. :-)
I was using diff and by eye balling the code, I couldn’t figure out why 2 lines were different. I used set list and saw the 2 lines each had a space at the end. I removed the spaces and diff was sucessful.
Helped me fix a strange problem that cropped up after I added some comments to a shell script !
Go figure.
Of course, if it is just the pesky ^M from a windows file that is the issue, there is always the good old “dos2unix” converston tool:
dos2unix -o a.xtx b.txt
That will automatically strip out the DOS line feed (carriage return code) stuff. HTH.
Another command to find odd and pesky unknown characters is to type “od -cb foobar” where foobar is your file name this will show the file with the character you see and the octal code for that character. The Octal Dump (od) program has many features and options, “man od” for more.
Brian, awesome tip! Thanks for sharing it. I need to compile some of the stuff people have added here to the post.
for a quick look at special chars, try this
:1,$l
the last character is an “ell” not a “one”
:1,$p prints the standard chars but not special
This is a great cheat sheet
http://www.lagmonster.org/docs/vi2.html
Mi, that’s awesome! Thanks for sharing.
dos2unix -n windows.txt unix.txt
I was just telling someone about this feature and I’m glad I found this post to share with them – thank you!
Additionally, you can define which hidden characters are displayed and how they appear.
I use Tim Pope’s Vim plugin vim-sensible, which sets the following.
set listchars=tab:>\ ,trail:-,extends:>,precedes:<,nbsp:+Thanks again for this post!
Hi! Thanks a lot for all the info!
I’m still trying to remove the $ that I’ve found… but I’ve tried :%s/$//g and didn’t work. What am I doing wrong?
Thanks!
P
Wow! 30 years of Unix/Linux and never knew that. My life just got slightly better.
Just what I needed a week ago before I rekeyed 4 script files. Now I KNOW I’m not crazy. No worries and and thanks to you that won’t happen again.